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United  States  General  Accounting  Office 
Washington,  D.C.  20548 


March  8,  2001 

The  Honorable  James  Inhofe 
Chairman 

The  Honorable  Daniel  Akaka 
Ranking  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

As  you  requested,  this  report  examines  how  best  practices  offer  improvements  to  the  way  the 
Department  of  Defense  defines  and  matches  weapon  system  requirements  to  available  resources  such 
as  cost,  schedule,  and  mature  technologies.  It  also  examines  the  importance  of  the  timing  of  this 
match  and  identifies  practices  that  can  improve  the  prospects  for  making  the  match  before  starting  an 
acquisition  program.  We  make  recommendations  to  the  Secretary  of  Defense  on  how  to  better  align 
the  requirements  setting  and  program  approval  processes  to  infuse  more  knowledge  earlier  into  each 
process. 

We  are  sending  copies  of  this  report  to  the  Honorable  Donald  H.  Rumsfeld,  Secretary  of  Defense;  the 
Honorable  Gregory  R.  Dahlberg,  Acting  Secretary  of  the  Army;  the  Honorable  Robert  B.  Pirie,  Jr., 
Acting  Secretary  of  the  Navy;  the  Honorable  Lawrence  Delaney,  the  Acting  Secretary  of  the  Air  Force; 
the  Honorable  Mitchell  E.  Daniels,  Jr.,  Director,  Office  of  Management  and  Budget;  and  to  interested 
congressional  committees.  We  will  also  make  copies  available  to  others  upon  request. 

If  you  have  any  questions  regarding  this  report,  please  call  me  at  (202)  5124841.  Other  key  contacts 
are  listed  in  appendix  D. 


Katherine  V.  Schinasi 
Director 

Acquisition  and  Sourcing  Management 
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Purpose 


Although  the  Department  of  Defense’s  (DOD)  annual  weapon  system 
investment  has  been  increased  from  about  $90  billion  3  years  ago  to  almost 
$100  billion  for  fiscal  year  2001,  DOD’s  buying  power  will  be  weakened  if 
weapons  continue  to  cost  significantly  more  and  take  much  longer  to 
develop  than  planned.  DOD  would  like  to  get  the  most  out  of  this 
investment  and  has  set  goals  to  develop  new  weapons  in  half  the  traditional 
time  and  within  budget.  It  has  a  long  way  to  go;  long-standing  problems  that 
work  against  delivering  new  weapons  within  estimates  have  proven 
resistant  to  reform.  When  one  program  encounters  such  problems  and 
needs  more  money  than  planned,  it  comes  at  the  expense  of  delaying  or 
canceling  other  programs.  This  means  less  overall  modernization  and  a 
lower,  unpredictable  return  on  investment.  The  ability  to  execute  a 
program  more  predictably  within  cost  and  schedule  estimates  would  lessen 
the  need  to  offset  cost  increases  by  disrupting  other  programs. 

GAO  has  issued  a  series  of  reports  on  the  success  leading  commercial  firms 
have  had  in  significantly  reducing  the  time  and  money  to  develop  new  and 
more  sophisticated  products— the  kinds  of  results  that  DOD  seeks.  The 
best  practices  of  these  firms  center  on  obtaining  knowledge  about  the 
technology,  design,  and  production  of  a  new  product  at  key  points  in  time. 
The  most  critical  juncture  is  the  decision  to  start  development  of  a  new 
product,  for  which  firms  insist  on  having  a  match  between  what  the 
customer  wants  in  a  new  product  and  what  resources  they  have  to  develop 
that  product.  Significant  cost  and  schedule  increases  in  weapon  system 
programs  can  be  traced  to  not  having  achieved  this  match  at  program  start. 

This  report  addresses  how  the  process  of  setting  requirements  for  new 
products  can  be  managed  in  a  way  that  does  not  exceed  the  developer’s 
resources  yet  provides  a  product  the  customer  wants.  In  response  to  a 
request  from  the  Chairman  and  the  Ranking  Member,  Subcommittee  on 
Readiness  and  Management  Support,  Senate  Committee  on  Armed 
Services,  GAO  assessed  (1)  the  effect  the  timing  of  the  match  between  the 
customer’s  needs  and  the  developer’s  resources  has  on  a  product’s  cost  and 
schedule;  (2)  the  best  practices  for  obtaining  this  match  during  the 
requirements  setting  process,  compared  with  more  traditional  DOD 
practices;  and  (3)  the  progress  made  and  challenges  DOD  faces  in  adopting 
best  practices  for  setting  requirements  on  individual  weapon  systems. 
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Background 


The  decisions  made  in  translating  the  ideas  for  a  new  product  into  actual 
features  and  characteristics  dictate  the  amount  of  resources  that  will  be 
necessary  to  bring  the  product  to  market.  Thus,  they  may  be  the  most 
highly  leveraged  of  all  product  development  decisions.  In  the  past,  it  has 
not  been  unusual  for  weapon  system  requirements  to  be  set  so  high  that  the 
initial  estimate  of  the  resources  necessary  to  develop  a  responsive  weapon 
falls  considerably  short  of  the  mark.  For  both  commercial  and  DOD 
products,  a  natural  amount  of  tension  precedes  the  setting  of  requirements, 
because  it  is  common  for  a  customer’s  initial  expectations  to  exceed  what 
the  developer  can  do  within  known  or  available  resources.  The  resources  a 
product  developer  can  consider  available  include  (1)  knowledge — the 
technology  and  capabilities  the  developer  has  to  engineer  and  manufacture 
the  product  and  (2)  the  time  and  money  the  developer  has  to  design,  test, 
and  deliver  the  product.  A  process  of  negotiation  and  trade-offs  is  usually 
necessary  to  match  customer  expectations  and  developer  resources  before 
a  product  can  be  successfully  developed  and  produced. 

Among  the  key  sources  of  information  GAO  relied  on  in  this  review  were 
experiences  from  nine  major  product  development  programs  from  DOD, 
the  National  Aeronautics  and  Space  Administration  (NASA),  and  two 
commercial  firms  recognized  for  their  success  in  setting  product 
requirements.  GAO  reviewed  the  requirements  setting  process  for  all  nine 
programs.  For  each  program,  GAO  interviewed  key  managers  and  obtained 
documentation  to  determine  (1)  the  process  that  was  used  to  achieve  the 
match  between  customer  expectations  and  resources  to  form  the  eventual 
product’s  requirements,  (2)  the  timing  of  this  match  and  the  tools  used  to 
achieve  it,  and  (3)  the  extent  to  which  the  requirements  setting  process 
affected  the  program’s  subsequent  progress  or  success.  While  the 
commercial  products  differed  significantly,  much  commonality  existed 
among  the  firms’  practices  for  managing  the  requirements  for  a  new 
product.  The  commercial  examples  in  the  report  were  chosen  for  their 
excellence;  as  such,  they  do  not  necessarily  represent  the  standard 
practices  of  commercial  industry  as  a  whole. 


Results  in  Brief 


A  match  between  a  developer’s  resources  and  a  customer’s  expectations  is 
eventually  met  on  just  about  every  product  or  weapon  system 
development.  A  key  distinction  between  successful  products — those  that 
perform  as  expected  and  are  developed  within  estimated  resources — and 
problematic  products  is  when  this  match  is  achieved.  When  a  customer’s 
needs  and  a  developer’s  resources  were  matched  before  a  product 
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development  started,  the  more  likely  the  development  was  to  meet  cost 
and  schedule  objectives.  When  this  match  took  place  later,  after  the 
product  development  was  underway,  problems  occurred  that  took 
significantly  higher  investments— sometimes  double— of  time  and  money. 

GAO  identified  three  factors  that  were  key  to  matching  needs  and 
resources  before  product  development  began.  First,  developers  employed 
the  technique  of  systems  engineering  to  identify  gaps  between  resources 
and  customer  needs  before  committing  to  a  new  product  development. 
Second,  customers  and  developers  were  flexible.  Leeway  existed  to  reduce 
or  defer  customer  needs  to  future  programs  or  for  the  developer  to  make 
an  investment  to  increase  knowledge  about  a  technology  or  design  feature 
before  beginning  product  development.  Third,  the  roles  and  responsibilities 
of  the  customer  and  the  product  developer  were  matched,  with  the  product 
developer  being  able  to  determine  or  significantly  influence  product 
requirements.  In  cases  where  these  factors  were  not  present  at  program 
launch,  product  development  began  without  a  match  between 
requirements  and  resources.  Invariably,  this  imbalance  favored  meeting 
customer  needs  by  adding  resources,  which  resulted  in  increased  costs  and 
later  deliveries. 

DOD  has  recently  revised  its  acquisition  policy  to  encourage  a  more 
evolutionary  approach  for  setting  requirements  and  developing  new 
weapons.  This  policy  merits  support;  if  effectively  implemented,  it  could 
facilitate  a  better  match  between  customer  needs  and  developer  resources 
before  program  launch.  DOD,  however,  faces  two  significant  hurdles  in 
implementing  the  policy.  First,  the  mechanics  of  obtaining  funds  to  start 
programs  are  unchanged.  Specifically,  requirements  must  still  be  set  before 
a  program  can  be  approved  and  a  program  must  be  approved  before  money 
can  be  paid  to  the  product  developer  to  conduct  systems  engineering.  Such 
mechanics  deny  the  knowledge  needed  to  match  customer  expectations 
with  developer  resources  before  starting  a  program.  Second,  many  of  the 
same  incentives  still  exist— such  as  the  competition  for  program  funding— 
to  push  requirements  up,  making  them  more  difficult  to  meet  and  less 
flexible  to  negotiate. 

GAO  makes  recommendations  to  the  Secretary  of  Defense  on  ways  to 
realign  the  mechanics  and  incentives  of  the  requirements  setting  and 
program  approval  process  with  the  need  to  match  customer  expectations 
and  developer  resources  before  weapon  system  programs  are  started. 
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Principal  Findings 


Timely  Match  of 
Requirements  and 
Resources  Is  Critical  to 
Product  Development 
Outcomes 


For  the  nine  development  programs  GAO  examined,  there  was  a 
relationship  between  when  customer  expectations  were  matched  with 
available  resources  and  the  programs’  ability  to  meet  cost  and  schedule 
predictions.  In  cases  where  needs  and  resources  were  matched  before 
program  start,  like  Caterpillar’s  797  mining  truck  and  NASA’s  Far 
Ultraviolet  Spectroscopic  Explorer,  cost  growth  was  20  percent  or  less  and 
product  development  schedules  were  met.  In  cases  where  programs  were 
started  with  requirements  that  exceeded  resources,  like  the  Crusader 
artillery  vehicle  and  the  Radio  Frequency  Countermeasures  system,  cost 
growth  ranged  from  55  percent  to  nearly  200  percent  and  schedule  delays 
were  about  25  percent.  Key  to  the  successful  cases  was  the  ability  to  make 
early  trade-offs  either  in  the  design  of  the  product  or  in  the  customer’s 
expectations  to  avoid  outstripping  the  resources  available  for  product 
development.  The  less  successful  cases  missed  opportunities  to  make 
trade-offs  before  product  development  started.  By  the  time  the  gaps 
between  requirements  and  resources  were  recognized  and  confronted,  it 
was  too  late  to  materially  change  the  requirements.  Consequently,  the  only 
route  left  was  for  the  developer  to  invest  more  resources  than  originally 
planned  to  meet  the  requirements. 


Several  Factors  Enable 
Customer’s  Needs  and 
Developer’s  Resources  to  Be 
Matched  Before  Program 
Launch 


GAO  found  significant  differences  between  the  successful  cases  and  those 
that  experienced  problems  regarding  (1)  how  systems  engineering  was 
employed,  (2)  how  flexible  customers  and  developers  were,  and  (3)  how 
balanced  the  roles  of  the  developer  and  the  customer  were.  Systems 
engineering  is  a  disciplined  process  that  translates  customer  needs  into  the 
specific  capabilities  that  are  needed  to  create  the  product,  such  as 
individual  technologies  and  manufacturing  processes.  It  is  critical  for 
identifying  and  resolving  gaps  between  needs  and  resources  and  lays  the 
factual  foundation  for  pragmatic  negotiations.  When  the  product  developer 
employed  systems  engineering  before  a  new  program  started,  the  resultant 
gaps  could  be  addressed  through  investments,  alternate  designs,  and 
trade-offs.  For  example,  Bombardier’s  systems  engineering  analysis  for  a 
new  regional-sized  jet  showed  that  if  the  customer  would  accept  a 
3.7-percent  reduction  in  cruise  speed,  existing  propulsion  technology  could 
be  used,  greatly  reducing  risk.  The  customer  agreed  to  the  reduction. 
Flexibility  was  encouraged  when  the  developer  set  a  limit  on  how  long  it 
would  allow  product  development  to  take  but  could  credibly  assure 


Page  9 


GAO-Ol-288  Best  Practices 


Executive  Summary 


customers  that  future  versions  of  a  product  would  meet  many  expectations 
deferred  from  the  initial  product.  Finally,  the  developer  and  the  customer 
were  equal  partners  in  setting  product  requirements;  a  new  product  was 
not  begun  unless  both  parties  agreed  to  the  requirements. 

In  the  cases  where  the  developers  did  not  deliver  the  products  as  promised, 
little  systems  engineering  had  been  done  by  the  time  requirements  were  set 
and  the  programs  were  launched.  The  bulk  of  systems  engineering — 
including  the  identification  of  gaps  between  resources  and  requirements — 
was  not  done  until  well  into  product  development.  For  example,  it  was 
2  years  after  the  Crusader’s  launch — after  requirements  were  set  and 
resource  estimates  made —  that  the  developer  concluded  that  a  key 
propellant  technology  could  not  be  developed  within  resources.  Also,  in 
these  cases,  the  customers  were  relatively  inflexible  regarding 
requirements  before  the  programs  were  launched.  Even  when  the  attempt 
was  made  to  deliberately  limit  the  length  of  the  product  development 
cycle — as  in  the  case  of  the  countermeasures  program — the  customer  was 
unwilling  to  make  compensating  trade-offs  in  performance.  Finally,  the 
customer  played  the  dominant  role  in  setting  requirements— a  process  that 
took  over  4  years  in  one  case — with  comparatively  little  input  from  the 
product  developer.  In  some  cases,  without  having  done  systems 
engineering,  the  product  developer  was  in  a  weak  position  to  disagree  with 
the  requirements.  In  other  cases,  the  product  developer  was  forced  to 
accept  the  requirements  despite  pointing  out  that  resources  were 
insufficient. 


Characteristics  of  DOD’s 
Acquisition  Process  Make  It 
Hard  to  Match  Needs  and 
Resources  Before  Program 
Launch 


In  DOD,  the  mechanics  of  obtaining  funding  and  getting  approval  to  start 
an  acquisition  program  dictate  that  events  proceed  in  the  following 
sequence:  set  requirements,  obtain  funding,  launch  program,  perform 
systems  engineering.  This  sequence  places  the  knowledge  that  is  needed  to 
identify  resource  gaps  and  shape  requirements  after  the  program  has  been 
launched  and  resource  estimates  have  been  made.  DOD  does  not  typically 
sign  contracts  with  product  developers  that  conduct  systems  engineering 
until  acquisition  programs  are  started  and  funding  is  received.  In  turn, 
programs  cannot  be  approved  unless  requirements  have  been  set.  By  the 
time  the  systems  engineering  is  started,  customers’  needs— as  well  as  those 
of  organizations  within  DOD  and  the  Congress  that  approve  and  fund 
programs — have  been  set,  making  it  difficult  to  change  requirements. 
Adding  resources — manifested  by  cost  and  schedule  increases — then 
becomes  the  primary  option  for  closing  gaps. 
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Recommendations  for 
Executive  Action 


DOD’s  process  for  setting  requirements  and  justifying  programs  creates 
incentives  for  setting  requirements  that  exceed  available  resources.  For 
example,  the  intense  competition  to  get  programs  approved  and  funded 
encourages  requirements  that  will  make  the  desired  weapon  system  stand 
out  from  others.  Also,  DOD  requirements  setters  are  often  motivated — not 
without  reason — to  aim  for  the  most  capability  possible,  given  that  it  may 
be  several  years  before  they  get  another  opportunity  for  a  new  weapon 
system  of  the  same  type.  Further,  they  do  not  necessarily  have  confidence 
that  DOD  will  be  able  to  fund  future,  more  capable  versions  of  a  weapon  if 
they  accept  minimum  capabilities  on  the  initial  version.  Finally,  the  DOD 
customer  is  more  willing  to  accept  cost  increases  and  schedule  delays  after 
program  launch  than  a  commercial  customer.  When  these  additional 
resources  are  provided  after  a  program  is  launched,  the  incentive  to  let 
requirements  drive  resources  up  is  reinforced. 

DOD  has  recently  adopted  policies  that  could  make  it  possible  for  a  better 
matching  of  customer  expectations  and  developer  resources  before 
program  launch.  These  policies  discourage  programs  from  accepting 
unreasonable  technical  risks  and  identify  ways  such  risks  can  be  reduced 
before  program  launch.  A  primary  way  is  the  policies’  expressed 
preference  for  an  evolutionary  approach  to  weapons  development  that 
calls  for  setting  a  reasonable — but  not  ultimate — requirement  for  an  initial 
version  of  a  weapon,  with  improvements  to  follow.  By  themselves, 
however,  the  policy  changes  do  not  materially  alter  the  mechanics  or  the 
incentives  that  shape  the  process  for  setting  requirements  and  justifying 
programs. 


GAO  recommends  that  the  Secretary  of  Defense  (1)  require  that  systems 
engineering  that  is  needed  to  evaluate  the  sufficiency  of  available  resources 
be  conducted  before  weapon  system  requirements  are  formalized, 

(2)  reduce  the  pressures  that  encourage  setting  high  and  inflexible 
requirements  to  win  the  competition  for  program  approval,  and  (3)  require, 
as  a  condition  for  starting  a  new  weapon  system  program,  that  sufficient 
evidence  exists  to  show  there  is  a  match  between  a  weapon  system’s 
requirements  and  the  resources  the  program  manager  has  to  develop  that 
weapon.  These  recommendations  appear  in  full  in  chapter  5. 
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Agency  Comments 


DOD  generally  agreed  with  the  report  and  its  recommendations.  A  detailed 
discussion  of  DOD’s  comments  appear  in  appendix  I. 
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For  several  years,  the  Department  of  Defense  (DOD)  has  expressed  an 
urgent  need  to  acquire  new  weapon  systems  to  replace  its  force  that  it 
believes  is  becoming  outdated  and  too  costly  to  operate.  DOD’s  annual 
weapon  system  investment  has  increased  from  about  $90  billion  3  years 
ago  to  almost  $100  billion  for  fiscal  year  2001;  over  the  next  5  years,  DOD 
plans  to  spend  about  $516  billion  developing  and  acquiring  weapon 
systems.  DOD  would  like  to  get  the  most  out  of  this  investment,  and  it  has 
set  goals  to  develop  new  weapons  in  half  the  traditional  time  and  within 
budget.  Historically,  DOD  has  not  received  a  predictable  return  on  its 
investment  in  major  weapon  systems  as  they  have  cost  significantly  more 
and  taken  much  longer  to  complete  than  originally  estimated.  When  one 
program  runs  into  problems  and  needs  more  money  than  planned,  it  comes 
at  the  expense  of  delaying  or  canceling  other  programs,  which  reduces 
buying  power  and  means  less  overall  modernization.  The  ability  to  execute 
a  program  more  predictably  within  cost  and  schedule  estimates  would 
lessen  the  need  to  offset  cost  increases  by  disrupting  other  programs.  DOD 
recognizes  that  changes  are  necessary  to  its  acquisition  practices  to 
achieve  its  modernization  goals.  Thus,  it  has  advocated  adopting  the 
practices  of  leading  commercial  firms. 

Our  reviews  over  the  past  20  years  have  likewise  pointed  to  a  need  to  adopt  ' 
new  practices.  We  have  seen  many  of  the  same  problems  recur  in  weapon 
system  programs— cost  increases,  schedule  delays,  and  performance 
problems.  On  many  occasions,  we  found  that  programs  required  more 
resources— time  and  money — than  were  estimated  for  demonstrating 
technologies,  designing  solutions,  and  providing  more  production 
capabilities  in  order  to  meet  customer  expectations.  Because  customer 
expectations  for  the  system’s  performance  were  set  when  the  decision  was 
made  to  invest  in  the  system,  adding  resources  became  the  primary  option  ■ 
for  solving  problems  when  they  arose.  Despite  good  intentions  and  some 
progress,  our  ongoing  reviews  of  DOD’s  weapon  system  acquisitions  show 
that  these  persistent  problems  remain.  As  a  result,  we  undertook  a  body  of 
work  that  examines  weapon  system  acquisitions  issues  from  a  different, 
more  cross-cutting  perspective — one  that  draws  lessons  learned  from  the 
best  commercial  product  development  efforts  to  see  if  they  can  be  applied 
to  DOD  weapon  system  developments.  In  past  years,  leading  commercial 
firms  have  developed  increasingly  sophisticated  products  in  significantly 
less  time  and  at  lower  costs.  1 

Our  past  work  has  shown  that  leading  commercial  firms  expect  their 
program  managers  to  deliver  high  quality  products  on  time  and  within 
budget.  Thus,  the  firms  have  created  an  environment  and  adopted  practices 
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that  put  their  program  managers  in  a  good  position  to  succeed  in  meeting 
these  expectations.  Collectively,  these  practices  comprise  a  development 
process  that  is  anchored  in  knowledge.  The  firms  demand — and  receive — 
specific  knowledge  about  a  new  product  at  key  junctures  in  the  process, 
(see  fig.  1). 


Figure  1 :  Knowledge-based  Process  for  Applying  Best  Practices  to  the  Development  of  New  Products 
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There  is  a  synergy  in  this  process,  as  the  attainment  of  each  successive 
knowledge  point  builds  on  the  preceding  one.  Such  a  knowledge-based 
process  is  essential  to  commercial  firms  getting  better — and  predictable — 
cost,  schedule,  and  performance  outcomes.  It  enables  decisionmakers  to 
be  reasonably  certain  about  critical  facets  of  the  product  under 
development  when  they  need  it.  We  have  found  that  when  DOD  programs 
have  employed  similar  practices,  they  also  experience  good  outcomes.  This 
knowledge  can  be  broken  down  into  three  knowledge  points: 

•  when  a  match  is  made  between  the  customer’s  needs  and  the  available 
technology; 

•  when  the  product’s  design  meets  performance  requirements,  and 

•  when  the  product  can  be  produced  within  cost,  schedule,  and  quality 
targets. 

The  most  important  knowledge  point  occurs  at  launch — the  point  at  which 
the  product  developer  makes  a  decision  to  commit  (or  invest)  the 
resources  necessary  to  develop  a  new  product  that  will  meet  customer 
needs.  This  knowledge  point  makes  it  easier  to  reach  the  remaining  two 
knowledge  points  at  the  right  time.  Successful  programs  are  launched  only 
when  a  product  developer  is  confident  that  it  has  the  resources — 
technology,  engineering,  and  production  knowledge,  along  with  sufficient 
time,  and  money — to  develop  a  product  the  customer  wants.  Significant 
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Is  Key  to  Program 
Outcomes 


problems  have  occurred  during  development  when  programs  were 
launched  without  this  match. 

We  have  reported  on  how  a  key  resource  of  a  developer — advanced 
technology — can  and  must  be  readied  to  meet  product  requirements  at  the 
time  a  product’s  development  program  is  launched.1  In  this  report,  we 
address  both  sides  of  the  match:  how  customer  needs  and  product 
developer  resources  can  be  managed  so  that  a  product  developer  can 
predictably  deliver  a  product  the  customer  wants. 


The  decisions  that  are  made  in  translating  the  ideas  for  a  new  product  into 
actual  features  and  characteristics  dictate  the  amount  of  resources — 
knowledge,  time,  money,  and  capacity — that  will  be  necessary  to  bring  the 
product  to  market.  Thus,  they  may  be  the  most  highly  leveraged  of  all 
product  development  decisions.  A  product’s  requirements  are  based  on 
customers’  expectations  and  justify  the  developer’s  investment  of 
resources  to  provide  the  desired  capability.  Requirements  drive  the  amount 
of  capital,  time,  expertise,  and  technologies  the  developer  must  invest.  In 
the  past,  it  has  not  been  unusual  for  weapon  system  requirements  to  be  set 
at  such  a  high  level  that  the  initial  estimate  of  the  resources  necessary  to 
develop  a  responsive  product  proves  insufficient,  evidenced  by  cost  growth 
and  schedule  slippage.  The  case  to  justify  the  requirements  is  often  so 
stridently  made  that  decisionmakers  are  in  a  relatively  weak  position  to  do 
anything  other  than  find  more  resources. 

For  commercial  firms  and  DOD,  the  basic  process  for  formulating  a 
product’s  requirements  is  the  same.  Each  begins  with  understanding  the 
customers’  expectations.  These  expectations  are  then  translated  into 
product  requirements  that  include  the  job  the  product  is  to  perform,  the 
functions  or  characteristics  it  is  to  possess,  the  practicality  it  must  have, 
and  its  reliability.  Typically,  the  first  understanding  of  customer 
expectations  exceeds  what  the  developer  can  do  within  available 
resources,  because  the  developer  has  a  limited  amount  of  resources  at  its 
disposal  for  product  development.  On  one  hand  is  knowledge — the 
technology  and  capabilities  the  developer  has  to  engineer  and  manufacture 
the  product.  On  the  other  hand  is  the  time  and  money  the  developer  has  to 
develop  additional  knowledge,  if  need  be,  and  to  design,  build,  test,  and 


1 Best  Practices:  Better  Management  of  Technology  Development  Can  Improve  Weapon 
System  Outcomes  (GAO/NSIAD-99-162,  July  30,  1999). 
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deliver  the  product.  It  is  not  unusual  for  a  customer  to  want  a 
high-performing  product  that  does  not  cost  much  or  take  too  long  to 
develop.  Such  an  expectation  may  exceed  the  developer’s  technology  or 
engineering  expertise  or  may  be  more  costly  and  time-consuming  to  create 
than  the  customer  is  willing  to  accept.  The  developer  must  stay  within  its 
means  if  the  venture  is  to  remain  mutually  beneficial.  Table  1  characterizes 
the  divergent  interests  of  the  customer  and  the  product  developer. 


Table  1:  Customer  and  Product  Developer  Interests  in  a  Product’s  Development 

Customer  wants 

Product  developer’s  resources 

Performance:  what  the  product  should  do. 
For  example,  what  an  aircraft’s  speed, 
range,  fuel  economy,  reliability,  and  other 
features  should  be. 

Technology:  the  technology  that  is  needed 
to  make  the  product  function  to  a  level 
necessary  to  meet  the  customer’s  wants. 

Timing:  when  the  customer  wants  the 
product. 

Schedule:  the  amount  of  time  that  is 
needed  to  develop,  design,  test,  and 
manufacture  the  product. 

Pricing:  what  the  product  will  cost.  The 
customer  must  be  able  to  afford  the  product. 

Investment  funds:  the  capital  that  is  needed 
to  pay  for  development,  test,  and 
manufacture  of  the  product  until  revenue 
from  sales  begins. 

Expertise:  the  capabilities  of  the  product 
developer  in  terms  of  engineering  expertise, 
manufacturing  capabilities,  and  production. 

Given  these  different  interests,  a  customer’s  wants  and  a  product 
developer’s  available  resources  must  be  matched  to  form  an  achievable  set 
of  product  requirements.  On  one  hand,  the  product  developer  must  develop 
and  produce  the  product  within  the  time  frames  the  customer  needs  or  the 
customer  may  find  an  alternative  product  or  source.  On  the  other  hand,  the 
customer  must  not  demand  a  product  that  requires  so  much  money  or  time 
to  develop  that  it  cannot  be  afforded  or  delivered  when  needed.  There  is  a 
delicate  balance  that  must  be  achieved  between  these  two  divergent 
interests  before  a  product  can  be  successfully  developed  and  produced. 

On  all  product  developments,  there  is  an  attempt  to  match  expectations 
with  available  resources  to  define  the  new  product.  A  customer’s 
expectations  and  a  product  developer’s  resources  are  more  closely 
scrutinized  during  the  matching  process  that  attempts  to  bring  the  two 
together.  The  outcome  is  a  set  of  product  requirements  that  represent  an 
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agreement  that  the  product  will  meet  the  customer’s  wants  and  that  the 
developer  can  deliver  the  product  within  acceptable  cost  and  schedule 
estimates.  The  requirements  then  guide  the  development  program.  This 
basic  requirements  setting  process  is  illustrated  in  figure  2. 


Figure  2:  The  Requirements  Process 


The  process  of  translat  ing  general  customer  expectations  into  a  specific  set 
of  product  requirements  involves  information  gathering,  analysis, 
negotiation,  and  agreement.  In  the  commercial  process,  the  customer  and 
the  product  developer  negotiate  requirements,  matching  expectations  and 
available  resources  into  a  documented  set  of  product  requirements  prior  to 
committing  resources  to  product  development.  During  this  negotiation,  the 
customer’s  relatively  unconstrained  wants  are  often  reduced  to  a  set  of 
performance  characteristics  that  are  achievable  with  available  resources, 
yet  still  meet  the  customer’s  needs.  The  commercial  process  is  a  two-way 
communication  between  the  customer  and  the  product  developer.  For 
example,  an  airline  company  may  want  a  certain  speed  to  maximize 
revenue  per  passenger  mile  from  a  new  aircraft.  However,  the  product 
developer  may  determine  that  the  resources  to  develop  an  aircraft  with  that 
speed  are  not  available  or  must  be  increased  dramatically.  Both  parties 
then  work  through  an  iterative  process  of  trades  and  negotiation  to  settle 
on  an  aircraft  with  mutually  acceptable  performance  and  resource 
requirements. 

The  DOD  process  is  somewhat  more  complex  and  involves 
communications  among  at  least  four  major  players.  On  one  end  is  the 
customer,  which  is  normally  a  military  organization  that  belongs  to  a  major 
fighting  force.  On  the  other  end  is  the  product  developer,  usually  a  defense 
firm  that  serves  as  the  prime  contractor  for  developing  and  producing  the 
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weapon  system.  In  between  are  two  other  players  that  actually  negotiate 
needs  and  resources  to  arrive  at  product  requirements.  One  is  referred  to 
as  the  user  representative,  which  is  an  organization  separate  from  the 
customer  but  represents  the  customer  and  negotiates  on  its  behalf.  The 
other  player  is  the  DOD  program  manager,  a  separate  organization  that,  in 
essence,  represents  the  product  developer.  Figure  3  illustrates  how  these 
different  players  interact  in  commercial  and  DOD  requirement-setting 
processes. 


Figure  3:  Commercial  and  DOD  Organizations  Involved  in  Requirements  Setting  Processes 


Commercial  example:  requirement  for  a  new  airliner 


Customer 


Product  developer 


DOD  example:  requirement  for  a  fighter  aircraft 


Customer  Government  Product 

Customer  representative  program  manager  developer 


Both  commercial  and  defense  organizations  are  concerned  about  how 
much  a  product  or  weapon  system  is  going  to  cost,  how  long  it  is  going  to 
take  to  build,  what  resources  will  be  needed  to  build  and  maintain  it,  and 
whether  it  works  properly.  All  of  these  concerns  are  translated  into  the 
product’s  requirements.  Unlike  the  commercial  process,  the  DOD  product 
developer  does  not  directly  influence  the  product  requirement  prior  to 
launching  product  development.  Once  requirements  are  formalized  in  what 
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DOD  refers  to  as  the  Operational  Requirements  Document,  they  are  turned 
over  to  the  prime  contractor,  who  actually  begins  product  development. 


Objectives,  Scope,  and 
Methodology 


The  Chairman  and  the  Ranking  Member,  Subcommittee  on  Readiness  and 
Management  Support,  Senate  Committee  on  Armed  Services,  requested 
that  we  examine  various  aspects  of  the  acquisition  process  to  determine 
whether  the  application  of  best  practices  can  improve  program  outcomes. 
To  date,  we  have  issued  reports  on  advanced  quality  concepts,  earned  value 
management,  management  of  a  product  from  development  to  production, 
management  of  key  suppliers,  management  of  technology  insertion, 
training,  and  management  of  test  and  evaluation  (see  related  GAO 
products). 


This  report  covers  the  beginning  of  the  acquisition  process:  the 
management  of  product  requirements.  Our  overall  objective  was  to 
determine  whether  best  practices  offer  methods  to  improve  the  way  DOD 
sets  product  requirements  within  the  framework  of  a  knowledge-based 
product  development  process.  Specifically,  we  assessed  (1)  the  effect  of 
the  timing  of  the  match  between  the  customer’s  needs  and  the  developer’s 
resources  on  a  product’s  cost  and  schedule;  (2)  the  best  practices  for 
obtaining  this  match  during  the  requirements  setting  process,  compared 
with  more  traditional  DOD  practices;  and  (3)  the  progress  made  and 
challenges  DOD  faces  in  adopting  best  practices  for  setting  requirements 
on  individual  weapon  systems. 


We  follow  a  similar  overall  methodology  for  conducting  best  practices 
reviews  in  the  area  of  weapon  system  development.  We  start  by  identifying 
individual  aspects  of  weapon  system  development — in  this  report,  the 
setting  of  requirements — that  have  been  shown  to  be  a  significant  and 
recurrent  cause  of  problems.  Our  sources  for  such  information  include  our 
many  reviews  of  individual  weapon  systems;  studies  from  other  sources, 
such  as  the  Defense  Science  Board;  and  discussions  with  defense  experts, 
including  past  and  current  DOD  officials,  defense  industry  representatives, 
and  analysts  from  private  organizations  that  study  defense  issues.  Before 
beginning  a  review  of  a  particular  topic,  we  confirm  with  DOD  officials  that 
the  topic  is  one  in  which  the  potential  for  improvement  is  significant.  Once 
we  have  identified  the  topic,  we  use  a  case  study  approach  because  case 
studies  provide  the  in-depth  knowledge  needed  to  understand  individual 
practices,  how  they  affect  program  outcomes,  and  why  they  are  adopted.  In 
selecting  case  studies,  we  look  for  examples  of  excellent  practices  from 
leading  commercial  firms,  examples  of  typical  or  traditional  practices  from 
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DOD,  and  where  possible,  DOD  examples  that  exhibit  excellent  practices. 
In  making  our  selections,  we  are  careful  to  make  sure  that  there  is  a  link 
between  the  practices  themselves  and  the  outcomes  of  the  programs. 

To  identify  best  practices  for  setting  requirements  on  commercial  products, 
we  reviewed  literature  and  spoke  with  industry  and  academic  experts  to 
find  companies  recognized  for  managing  requirements  to  help  deliver  new 
products  that  were  both  quicker  to  market  and  more  advanced  than  their 
predecessors.  We  identified  three  companies 

•  Caterpillar  Construction  and  Mining  Division,  Decatur,  Illinois; 

•  Bombardier  Aerospace,  Toronto,  Canada;  and 

•  Bethlehem  Steel,  Bethlehem,  Pennsylvania 

We  visited  each  company,  discussed  the  process  used  for  setting 
requirements,  and  obtained  an  understanding  of  the  overall  process  used 
with  emphasis  on  those  practices  each  felt  were  critical  for  success.  We 
also  met  with  individual  program  managers  and  discussed  specific  product 
development  examples  that  further  illustrated  the  process.  During  our 
discussions  with  the  firms,  we  compared  and  contrasted  the  best  practices 
with  DOD’s  practices. 

We  developed  nine  case  studies  in  total.  These  included  two  commercial 
case  studies,  one  National  Aeronautics  and  Space  Administration  (NASA) 
program  that  had  received  excellent  results  by  disciplining  its 
requirements-setting  process,  and  six  DOD  weapon  system  programs  that 
represented  a  mixture  of  traditional  and  best  practices.  We  reviewed  the 
requirements-setting  process  for  all  nine  programs.  For  each  program,  we 
interviewed  key  managers  and  obtained  documentation  to  determine 
(1)  the  process  that  was  used  to  achieve  the  match  between  customer 
wants  and  resources  to  form  the  product’s  requirements,  (2)  the  timing  of 
this  match  and  the  tools  used  to  achieve  it,  and  (3)  the  extent  to  which  the 
requirements  setting  process  affected  the  program’s  product  development 
outcome.  Descriptions  of  the  nine  programs  we  reviewed  follow. 

Commercial 

•  Bombardier’s  BRJ-X,  a  commercial  jet  in  development,  designed  to  carry 
between  88  to  110  passengers.  It  bridges  the  gap  between  the  current 
fleet  of  regional  jets,  20  to  70  passenger  capacity  and  the  larger  111  to 
170  passenger  commercial  airlines.  The  BRJ-X  program  was  launched  in 
the  second  quarter  of  2000.  The  first  aircraft  is  scheduled  to  be  delivered 
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to  airlines  in  late  2003.  Bombardier  estimates  customers  for  2,500 
aircraft  over  the  next  20  years. 

•  The  Caterpillar  797  mining  truck,  the  largest  mining  truck  ever  built.  It 
can  carry  over  360  tons  of  ore,  and  features  many  patented  innovations. 
Developed  in  response  to  mining  companies’  desire  to  reduce  cost  per 
ton  of  hauling  ore  in  large-scale  mining  operations,  Caterpillar  launched 
the  797  program  in  1997. 

DOD 

•  The  Air  Force’s  Global  Hawk,  an  unmanned  aircraft  that  is  intended  to 
fly  at  altitudes  as  high  as  65,000  feet  and  for  as  long  as  40  hours  to 
provide  the  Air  Force  with  an  intelligence,  reconnaissance,  and 
surveillance  capability.  The  Air  Force  built  five  prototype  technology 
demonstrators  that  were  used  to  demonstrate  the  aircraft,  and  it  plans 
to  launch  the  product  development  program  in  2001. 

•  The  Army’s  Crusader  artillery  vehicle  program,  a  self-propelled 
155-millimeter  howitzer  and  resupply  vehicle.  It  is  expected  to  be  the 
first  fully  automated,  computerized,  and  tracked  artillery  system.  The 
Crusader  development  program  began  in  1994,  and  the  howitzer  is 
expected  to  start  production  in  2008.  The  development  program  is 
estimated  to  cost  $4.3  billion. 

•  The  Army’s  Tactical  Unmanned  Aerial  Vehicle,  a  short-range  unmanned 
aircraft  that  is  expected  to  provide  the  Army  with  day  or  night 
reconnaissance,  surveillance,  and  target  acquisition  capability.  The 
Army  began  development  in  March  1999.  It  plans  to  buy  44  systems 
starting  in  2001.  Each  system  includes  three  unmanned  aircraft;  a 
vehicle  to  carry  the  aircraft;  two  ground  control  stations  mounted  on 
vehicles;  and  launch,  recovery,  and  support  equipment  pulled  on  trailers 
behind  the  vehicles.  The  cost  to  buy  the  44  systems  is  estimated  at 
$430  million  through  2004. 

•  The  Navy’s  Radio  Frequency  Countermeasures  system,  an  electronics 
warfare  system  that  uses  a  jamming  device  called  a  techniques 
generator.  It  is  carried  onboard  an  aircraft  to  produce  jamming  signals 
that  are  transmitted  by  fiber  optic  cable  to  a  towed  device  that  acts  as  a 
decoy  for  the  aircraft.  The  system  is  used  to  protect  the  aircraft  from 
radar-controlled  weapons  like  missiles  and  antiaircraft  artillery.  It  is  a 
critical  component  of  the  Integrated  Defensive  Electronics 
Countermeasure  System  being  developed  for  some  Navy  and  Air  Force 
aircraft.  System  development  began  in  1995  and  is  expected  to  cost  over 
$200  million.  It  is  expected  to  enter  production  in  2002. 


Page  22 


GAO-0 1-288  Best  Practices 


Chapter  1 
Introduction 


•  The  Army’s  Brilliant  Anti-Armor  Submunition  program,  referred  to  as 
BAT,  an  acoustic  and  infrared  terminally  guided  submunition  that 
searches  for,  detects,  tracks  and  engages  moving  tanks  and  armored 
combat  vehicles.  Its  mission  is  to  provide  deep  attack  against  motorized 
rifle  and  tank  divisions.  The  carrier  for  the  submunition  is  the  Army 
Tactical  Missile  System  that  is  launched  from  the  Multiple  Launch 
Rocket  System.  The  Army  plans  to  buy  15,707  submunitions  at  a  cost  of 
$2.5  billion. 

•  The  Army’s  Comanche  helicopter,  a  lightweight,  twin  engine,  stealthy 
helicopter  that  is  intended  to  replace  the  Army’s  OH-58  and  AH-1 
helicopters.  The  primary  mission  of  the  aircraft  will  be  armed 
reconnaissance  and  attack.  It  is  the  Army’s  largest  aviation  acquisition 
program,  with  a  projected  total  development  and  production  cost  of 
$48  billion  for  1,213  helicopters.  The  development  program  was 
launched  in  1988,  and  production  is  expected  to  start  in  2006. 

NASA 

•  The  Far  Ultraviolet  Spectroscopic  Explorer  (FUSE),  a  scientific 
telescope  that  is  used  for  studying  the  origin  and  evolution  of  stars, 
galaxies,  and  planetary  systems.  The  telescope  is  18  feet  tall  and  weighs 
3,000  pounds.  It  was  developed  for  NASA  by  the  John  Hopkins 
University.  The  FUSE  program  began  in  June  1996  and  was  completed  in 
June  1999  at  a  cost  of  $120  million. 

We  used  information  from  our  prior  best  practices  work,  including  most  of 
the  information  on  the  BAT  and  Comanche  programs.  Similarly,  we 
gathered  knowledge  about  many  aspects  of  the  product  development 
processes,  including  the  setting  of  requirements,  from  leading  commercial 
firms  in  addition  to  the  firms  included  in  this  report.  During  the  past 
4  years,  we  have  gathered  information  on  product  development  practices 
from  3M,  Boeing  Airplane  Company,  Chrysler  Corporation,  Ford  Motor 
Company,  Hughes  Space  and  Communications,  and  Motorola.  This 
information  enabled  us  to  develop  an  overall  model  to  describe  the  general 
approach  leading  commercial  firms  take  to  developing  new  products. 

We  also  met  with  experts  in  the  area  of  setting  product  requirements  from 
academia  and  participated  in  conferences  and  workshops  with  recognized 
leaders  in  the  acquisition  field  to  obtain  information  on  how  organizations 
were  improving  their  acquisition  processes.  To  obtain  a  general 
understanding  of  DOD’s  requirements  setting  process  and  improvement 
initiatives,  we  met  with  officials  from  the  Office  of  the  Secretary  of 
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Defense;  Army,  Navy,  and  Air  Force  Headquarters;  and  the  Joint  Staff  for 
the  Joint  Chiefs  of  Staff.  We  also  had  discussions  with  former  DOD  officials 
and  industry  experts  about  DOD  acquisition  policies  and  practices.  With 
these  officials  we  discussed  the  current  process,  initiatives  and  the 
applicability  of  best  practices  to  DOD  operations.  In  addition,  we  visited 
NASA  to  obtain  information  on  their  processes  and  practices  for  setting 
requirements  for  new  product  development  programs. 

We  conducted  our  review  from  November  1999  through  December  2000  in 
accordance  with  generally  accepted  government  auditing  standards. 
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The  point  in  time  that  a  developer  for  a  new  product  becomes  justifiably 
confident  that  it  has  the  resources — knowledge,  capacity,  time,  and 
money — to  develop  a  product  that  a  customer  wants  is  critical  to  the 
success  of  the  development  effort.  Barring  program  cancellation,  the  match 
between  resources  and  wants  is  eventually  met  on  just  about  every  product 
or  weapon  system  development.  A  key  distinction  between  successful 
products — those  that  perform  as  expected  and  are  developed  within 
estimated  resources — and  problematic  products  is  when  this  match  is 
achieved.  Simply  put,  we  found  that  when  wants  and  resources  were 
matched  before  a  product  development  was  started,  the  more  likely  the 
development  was  able  to  meet  performance,  cost,  and  schedule  objectives. 
When  this  match  took  place  later,  programs  encountered  problems  such  as 
increased  costs,  schedule  delays,  and  performance  shortfalls. 

For  successful  product  or  weapon  system  development  program  cases, 
trade-offs  were  made  either  in  the  design  of  the  product  or  in  the 
customer’s  expectations  to  avoid  immature  technologies  or  exotic 
components  that  threatened  to  outstrip  the  developer’s  resources.  In  the 
less  successful  cases,  the  opportunity  to  make  such  trade-offs  before 
starting  product  development  was  missed  either  because  gaps  between 
expectations  and  resources  were  not  identified  or  because  customers  were 
unwilling  to  reduce  their  expectations.  When  the  divergence  between 
customer  expectations — which  by  then  had  become  firm  product 
requirements — and  developer  resources  was  recognized  and  confronted, 
decisionmakers  were  reluctant  to  materially  change  the  requirements. 
Consequently,  the  developer  had  to  invest  more  resources  than  originally 
planned  to  meet  the  requirements. 


Early  Matching  of 
Customer 
Expectations  and 
Developer’s  Resources 
Is  Critical  to  Program 
Success 


We  found  that  the  timing  of  the  matching  process — when  the  customer’s 
expectations  were  successfully  matched  with  the  product  developer’s 
resources — significantly  influenced  the  likely  success  of  a  product’s 
development.  After  reviewing  the  process  for  defining  product 
requirements  for  nine  development  programs,  we  found  a  relationship 
between  when  expectations  were  matched  with  available  resources  and 
when  the  cost  and  schedule  predictions  for  the  programs  were  achieved. 
This  relationship  is  shown  in  table  2. 
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Table  2:  Matching  of  Expectations  to  Resources  and  Product  Development  Outcomes 


Programs 

Expectations  and  resources 
adequately  matched  before 
launch 

Product  development 
cost  growth 

Product  development 
schedule  delays 

Caterpillar  797  mining  truck 

Yes 

5  percent 

0  percent 

NASA  FUSE 

Yes 

20  percent 

0  percent 

Radio  Frequency  Countermeasures 
system 

No 

197  percent 

23  percent 

Crusader  artillery  vehicle 

No 

55  percent 

26  percent 

Comanche  helicopter 

No 

127  percent 

1 1 9  percent 

Brilliant  Anti-armor  Submunition 

No 

99  percent 

46  percent 

Bombardier  BRJ-Xa 

Yes 

On  target 

On  target 

Tactical  Unmanned  Aerial  Vehicle3 

Yes 

On  target 

On  target 

Globa!  Hawk  Unmanned  Vehicle8 

Yes 

On  target 

On  target 

a  Specific  cost  and  schedule  data  for  the  Tactical  Unmanned  Aerial  Vehicle,  Global  Hawk  Unmanned 
Aerial  Vehicle,  and  Bombardier  BRJ-X  Regional  jet  were  not  included  in  the  table  because  they  had 
not  been  in  the  product  development  phase  long  enough  to  report  actual  cost  and  schedule  variances. 
However,  these  programs  had  already  avoided  some  of  the  problems  experienced  by  the  programs 
that  did  not  match  expectations  and  resources  before  launch.  These  programs  were  on  target  for 
meeting  their  objectives. 

The  more  successful  programs  had  matches  before  the  commitment  to 
launch  the  programs  was  made.  In  each  case,  the  product  developer  had 
done  the  initial  design  of  the  product  and  ensured  that  only  proven 
technologies,  design  features,  and  production  processes  would  be  used. 
This  was  accomplished  by  either  making  additional  investments  to 
demonstrate  uncertainties  such  as  new  technology  or  by  reducing  the 
product’s  initial  performance  requirements.  These  steps  maximized  the 
knowledge  content  of  the  product  and  enabled  the  program  manager  to  set 
cost  and  schedule  estimates  it  could  reasonably  expect  to  meet.  In 
contrast,  the  programs  that  did  not  have  matches  before  launch  did  so 
during  product  development  by  the  unplanned  addition  of  resources.  This 
contributed  significantly  to  cost  and  schedule  problems.  Figure  4  illustrates 
the  timing  of  the  match  between  customer’s  expectations  and  a  product 
developer’s  resources. 
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Figure  4:  Timing  of  Match  Between  Customer  Expectations  and  Resources 


matched  customer  development  by  investing 

expectations  with  additional  resources 

developer’s  resources 


The  programs  in  which  expectations  and  resources  were  matched  before 
product  development  started  were  in  a  good  position  to  commit  to  cost  and 
schedule  estimates  that  were  attainable.  In  those  cases  where  expectations 
and  resources  were  not  matched  before  launch,  cost  and  schedule 
estimates  had  to  be  made  at  the  time  of  launch.  Such  estimates  were 
necessarily  made  at  levels  consistent  with  the  resources  the  product 
developer  had  available,  under  the  assumption  that  either  (1)  no  gaps 
existed  between  expectations  and  resources  or  (2)  any  gaps  could  be 
closed  within  projected  resources.  Because  customer  expectations 
regarding  the  performance  of  the  product  tended  to  become  set  when 
product  development  began,  adding  resources  emerged  as  the  primary 
option  available  to  match  expectations  and  resources.  These  resources 
(time  and  money)  were  typically  needed  for  maturing  technologies, 
developing  design  solutions,  and  providing  more  production  capabilities. 
Perhaps  more  importantly,  they  were  not  estimated  or  planned  for  and 
often  necessitated  sacrifices  in  other  needs,  such  as  reducing  the  resources 
of  other  development  programs. 
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Trade-offs  Are  Critical 
to  Matching  Customer 
Expectations  With 
Developer  Resources 
Before  Starting 
Product  Development 


Caterpillar  and  Bombardier  both  matched  customer  expectations  with 
available  resources  prior  to  setting  product  requirements  and  launching  the 
products’  development  programs.  In  each  case,  there  were  differences 
between  expectations  and  resources  that  necessitated  trade-offs  before 
requirements  could  be  set  and  the  product’s  development  could  be 
launched.  For  these  cases,  expectations  and  resources  were  negotiated  so 
that  the  product’s  requirements  could  be  achieved  within  available 
resources  while  still  meeting  the  customers’  critical  needs.  This  allowed 
the  firms  to  develop  and  deliver  their  products  quickly  and  within 
acceptable  cost  limits,  thereby  maintaining  their  competitive  advantage  in 
their  respective  markets. 


Caterpillar’s  797  Mining  To  maintain  the  competitive  advantage  in  its  market,  Caterpillar’s  board  of 

'JYuck  directors  believed  it  had  to  develop  by  the  end  of  1998  a  new  product  that 

was  sized  to  work  efficiently  with  large  loading  shovels  used  in  mining 
operations.  This  meant  that  Caterpillar’s  product  development  team  would 
have  about  18  months  from  the  time  the  product  development  program  was 
approved  to  develop  and  field  the  797,  a  newly  designed  truck  that  could 
efficiently  haul  at  least  360  tons  of  payload.  According  to  Caterpillar,  they 
met  this  date  because  they  made  trade-offs  between  the  customers’ 
expectations  and  the  resources  available  before  the  product  development 
program  began.  Figure  5  shows  the  797  mining  truck  developed  and 
figure  6  shows  the  history  of  the  track  development. 


Page  28 


GAO-Ol-288  Best  Practices 


Chapter  2 

Timely  Matching  of  Requirements  and 
Resources  Is  Critical  to  Product 
Development  Outcomes 


Figure  5:  Caterpillar  Mining  Truck 


Caterpillar  developed  the  797,  a  360-ton  capacity  mining  truck,  in  18  months. 
Source:  Caterpillar,  Inc. 


\m 


December 

1998 


Caterpillar  determined 
that  there  is  a  customer 
need  for  ultra  class 
truck. 


Start  of  product 
development  delayed 
because  requirements  did 
not  match  resources. 
Trade-offs  needed  to 
achievel  8-month  schedule. 


Program  launched  after 
requirements  that  could 
not  be  met  with  available 
technologies  were 
deferred  to  future 
versions  of  the  797. 


Caterpillar  met  the 
1 8-month  schedule  and 
was  within  5  percent 
of  its  cost  objective. 
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Examples  of  key  trade-offs  Caterpillar  made  to  close  gaps  between 

customer  expectations  and  its  own  resources  were: 

•  deferring  to  the  next  product  line,  new  prognostic  technologies  wanted 
by  the  customer  that  could  assist  in  forecasting  wear  and  tear  to  the 
truck,  but  were  immature  at  the  time  product  development  started; 

•  selecting  a  twin-engine  propulsion  design  to  power  the  797  rather  than  a 
single  engine — despite  its  potential  for  lower  operating  costs — because 
it  had  not  yet  been  developed  and  was  therefore  too  much  of  a  risk;  and 

•  redesigning  the  wheel  and  transmission  to  avoid  the  need  to  develop 
new  gears  for  the  differential  unit  in  the  drivetrain,  which  avoided  a 
costly  and  risky  development  effort  that  could  have  impacted  the  truck’s 
development  progress. 


The  Bombardier  BRJ-X  Jet  During  the  BRJ-X  jet’s  concept  definition  phase,  which  preceded  product 

development,  Bombardier  identified  some  customer  expectations  that  its 
designers  believed  would  put  program  cost  and  schedule  at  risk.  The 
expectations  were  analyzed  and  trade-offs  were  made  to  make  the  design 
achievable  within  resources.  For  example,  both  airline  customers  and 
Bombardier  wanted  to  use  fly-by-wire  flight  control  technology,  which 
would  replace  heavy  hydromechanical  flight  controls  with  lighter  weight 
electronic  controls,  on  the  BRJ-X  to  reduce  weight  and  lower  fuel  costs. 
However,  Bombardier  engineers  had  some  concerns  about  the  technology 
since  it  had  never  been  integrated  into  a  regional  jet  configuration  before. 
Consequently,  Bombardier  decided  to  invest  the  time  and  money  to 
demonstrate  the  technology  on  a  surrogate  business  jet  before  making  it  a 
program  requirement.  Another  trade-off  was  made  after  Bombardier 
determined  that  the  desired  speed  would  require  design  effort  and  features 
not  fully  proven  out.  The  customers  agreed  that  a  slight  reduction  in  speed 
would  eliminate  the  design  concerns  yet  still  meet  their  expectations. 


NASA’s  FUSE  Program  FUSE  is  a  telescope  used  to  study  the  origin  and  evolution  of  certain 

elements  in  space  to  help  determine  the  age  and  evolution  of  stars, 
galaxies,  and  planetary  systems.  The  customer  initially  expressed 
expectations  in  terms  of  gathering  precise  images  of  light  from  these 
elements  in  space  24  hours  per  day.  When  NASA  threatened  to  cancel  the 
program  because  requirements  were  not  achievable  given  available 
resources,  FUSE  program  managers  negotiated  with  customers  and  found 
they  could  still  meet  the  basic  expectations  with  reduced  requirements  for 
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a  high-resolution  mirror,  bandwidth  detection,  and  time  on  orbit;  switching 
from  a  highly  elliptical  orbit  to  a  low  earth  orbit  (see  fig.  7). 


Figure  7:  Effect  of  FUSE  Trade-offs  on  Matching  Customer  Expectations  with  Developer’s  Resources 


Customer’s  expectations  Lower  mirror  resolution 


These  reductions  in  requirements  allowed  the  use  of  existing  technologies 
such  as  a  new  grating  technique  to  spread  radiation  into  different 
wavelengths.  Matching  requirements  to  resources  allowed  FUSE  to  not 
only  meet  schedule  targets  and  be  within  20  percent  of  its  cost  objective 
but  also  a  critical  NASA  need. 


When  Matching  Did 
Not  Occur  Before 
Program  Launch, 
Developers  Were 
Forced  to  Add 
Unplanned  Resources 


In  cases  where  the  customers’  expectations  were  not  matched  with  the 
developers’  resources  before  product  development  began,  the  matches 
took  place  after  opportunities  to  make  trade-offs  had  passed.  Because  new 
programs  had  been  approved  with  customer  wants  formally  documented  as 
requirements,  the  main  avenue  available  to  close  gaps  between 
requirements  and  resources  was  for  the  developers  to  invest  more  effort, 
time,  and  money  to  gain  the  knowledge  and  capacity  needed  to  meet  the 
requirements.  This  additional  investment — manifested  by  cost  increases 
and  schedule  delays — was  not  planned,  which  forced  trade-offs  or  cuts  to 
be  made  in  other  programs  to  free  up  the  additional  resources.  In  some 
cases,  the  developers  may  have  had  indications  that  there  was  a  mismatch 
at  the  launch  decision,  but  they  were  pressured  to  go  forward  anyway.  In 
other  cases,  while  the  developers  may  not  have  enough  knowledge  to  be 
confident  that  there  was  a  match  before  launch,  they  were  at  a 
disadvantage  to  argue  that  there  was  not.  DOD’s  Radio  Frequency 
Countermeasures  system  and  Comanche  helicopter  programs  did  not  have 
matches  before  their  product  development  programs  were  launched.  In 
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fact,  despite  being  several  years  into  both  programs,  it  is  still  uncertain 
whether  this  match  has  been  reached. 


Navy’s  Radio  Frequency  Customer  expectations  and  developer  resources  were  not  matched  on  the 

Countermeasures  System  Radio  Frequency  Countermeasures  program  when  product  development 

began  in  1995.  Essentially,  the  performance  wanted  by  the  customers 
exceeded  the  time,  money,  and  technologies  available  to  the  product 
developer  to  develop  an  acceptable  system.  For  example,  a  critical 
component  is  the  fiber-optic  towed  decoy — a  component,  towed  behind  the 
aircraft  in  flight,  which  transmits  electronic  countermeasures.  Before 
program  launch,  the  requirement  for  the  power  to  be  transmitted  from  the 
towed  decoy  was  nearly  tripled  because  of  a  last  minute  addition  to  satisfy 
the  Air  Force’s  expectations  for  the  F-15  fighter  and  the  B-l  bomber.  This 
was  a  demanding  requirement  that,  according  to  the  former  program 
manager,  required  some  technological  invention.  According  to  the  manager, 
this  was  complicated  by  the  customer’s  refusal  to  either  reduce 
performance  requirements  or  accept  increases  in  time  and  cost  to  develop 
a  product  that  would  meet  these  requirements.  The  Navy  chose  to  launch 
the  program  with  this  mismatch,  recognizing  that  it  added  risk  to  the 
program.  Figure  8  shows  the  timeline  for  the  program. 


Figure  8:  Navy’s  Development  of  the  Radio  Frequency  Countermeasures  System 


As  the  program  proceeded,  problems  associated  with  the  risk 
accompanying  the  mismatch  turned  into  development  problems,  and  the 
additional  resources  found  to  be  unacceptable  to  the  customer  before 
launch  were  accepted  as  a  necessity  after  launch.  Consequently,  the  costs 
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rose  from  $74  million  to  $221  million  and  the  development  schedule  was 
extended  15  months. 


Comanche  Helicopter  Customer  expectations  that  became  key  requirements  for  the  Comanche 

helicopter  demanded  several  technologies  that  were  still  very  immature 
when  the  Army  decided  to  launch  the  program  in  1988.  For  example,  the 
integrated  avionics  and  an  advanced  infrared  night  vision  and  targeting 
sensor  were  included  on  the  program  when  they  were  still  conceptual  in 
nature.  These  advanced  avionics  systems  were  needed  to  meet  the 
customer’s  “must-have”  requirements  for  a  very  lightweight,  stealthy,  highly 
maneuverable,  all-weather  reconnaissance  and  attack  helicopter.  These 
technologies  were  also  critical  to  meet  cost  and  weight  goals  for  the 
program.  The  Army  launched  the  program  despite  the  low  readiness  of  the 
technologies,  with  the  developer  having  limited  design  alternatives  but 
believing  that  the  needed  technological  invention  could  be  accomplished 
within  projected  resources.  At  the  time  of  launch,  the  Army  estimated  that 
product  development  would  cost  $3.6  billion  and  last  about  8  years.  Due  to 
problems  that  developed  with  these  technologies  and  budgeting  and  other 
changes  in  the  program,  development  is  now  estimated  to  take  $8.3  billion 
and  18  years — over  100-percent  increases.  The  Army  kept  the  customer’s 
requirements  essentially  unchanged,  electing  to  double  the  resources 
needed  to  meet  them. 
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The  ability  to  match  a  customer’s  wants  with  resources  before  launching  a 
development  program  is  key  to  putting  program  managers  in  a  better 
position  to  succeed.  We  found  three  factors  that  comprise  this  ability.  First, 
developers  employed  systems  engineering  to  identify  gaps  between 
resources  and  customer  wants  before  committing  to  a  new  product 
development.  Second,  customers  and  product  developers  were  flexible 
before  launch.  Leeway  existed  to  reduce  expectations,  defer  them  to  future 
programs,  or  to  invest  more  resources  up  front  to  eliminate  gaps  between 
resources  and  expectations.  Third,  the  roles  and  responsibilities  of  the 
customer  and  the  product  developer  were  balanced,  with  the  product 
developer  given  the  responsibility  to  determine  or  significantly  influence 
product  requirements.  In  cases  where  these  factors  were  not  present  at 
program  launch,  product  development  began  with  imbalanced  product 
requirements.  Invariably,  this  imbalance  favored  meeting  expectations  at 
the  expense  of  resources,  putting  the  developers  at  a  disadvantage  to 
deliver  the  products  within  cost  and  schedule  estimates. 

In  the  most  successful  cases,  the  effective  interplay  of  these  factors 
allowed  the  customers  and  product  developers  to  arrive  at  a  set  of  product 
requirements  that  could  be  developed  within  cost  and  schedule  targets. 
Systems  engineering  provided  knowledge  necessaiy  to  translate  customer 
wants  into  specific  capabilities,  enabling  the  developers  to  identify  and 
resolve  gaps  before  product  development  began.  With  systems  engineering 
knowledge  in  hand,  flexible  requirements  were  essential  to  lowering  risk 
through  negotiations  because  knowledge  alone  did  not  produce  trade-offs. 
Absent  such  flexibility,  resources  and  wants  could  still  be  matched  before 
product  development  began,  but  the  options  to  resolving  any  gaps  were 
limited  to  additional  investments  on  the  developers’  part.  Finally,  with 
knowledge  gained  from  systems  engineering  and  flexible  requirements  as 
preconditions,  successful  product  development  programs  benefited  from 
an  environment  in  which  product  developers  and  customers  had  balanced 
roles  and  shared  responsibilities  for  setting  product  requirements. 
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Systems  Engineering 
Tools  Are  Critical  for 
Identifying  Gaps 
Between  Developer’s 
Resources  and 
Customer’s 
Expectations 


When  product  developers  employed  systems  engineering  before 
committing  to  product  development,  they  were  able  to  identify  areas  in 
which  the  customers’  wants  exceeded  their  resources.  For  those  cases  in 
which  developers  did  not  conduct  sufficient  systems  engineering  before 
committing  to  the  new  product  development,  they  were  weakened  in  their 
ability  to  identify  gaps  between  their  resources  and  expectations.  These 
gaps  were  later  revealed  as  unexpected  problems  that  required  invention, 
time,  and  money  to  resolve. 

Systems  engineering  is  a  process  that  not  only  translates  customer  wants 
into  specific  capabilities,  such  as  individual  technologies  and 
manufacturing  processes,  but  also  provides  knowledge  that  enables  a 
developer  to  identify  and  resolve  gaps  before  product  development  begins. 
It  is  defined  as  a  logical  sequence  of  activities  that  transforms  a  customer 
want  into  specific  product  characteristics  and  functions  and  ultimately  into 
a  preferred  design  (see  fig.  9).  It  is  not  necessarily  the  use  of  systems 
engineering  in  the  development  of  a  new  product  or  weapon  system,  but 
when  it  is  used  that  distinguishes  it  as  a  best  practice. 


Figure  9:  Basic  Steps  in  Systems  Engineering  Process 


The  systems  engineering  discipline  enables  the  product  developer  to 
translate  customer  wants  into  specific  product  features  for  which  requisite 
technological,  software,  engineering,  and  production  capabilities  can  be 
identified.  Once  these  capabilities  are  identified,  a  developer  can  assess  its 
own  capabilities  to  determine  if  gaps  exist.  It  is  critical  for  a  developer  to 
involve  the  right  people — those  with  the  affected  areas  of  expertise — in 
this  assessment.  Gaps  identified  between  what  the  customer’s  wants  are 
and  what  the  developer  possesses  then  become  the  focus  of  analysis.  Some 
gaps  can  be  resolved  by  investments  the  developer  makes,  while  others  can 
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be  closed  by  finding  technical  or  design  alternatives.  Remaining  gaps — 
those  that  represent  capabilities  the  developer  does  not  have  or  cannot  get 
without  increasing  the  price  and  timing  of  the  product  beyond  what  the 
customer  will  accept — must  be  resolved  through  trade-offs  and 
negotiation. 

During  systems  engineering,  a  product  design  progresses  through  at  least 
three  iterations.  The  first  is  a  notional  design — a  general  concept  of  what 
the  product  will  look  like  and  what  it  might  be  capable  of  that  is 
unconstrained  by  resources.  The  second  iteration  is  the  first  detailed 
design  that  enables  a  developer  to  compare  its  capabilities  with  the 
demands  of  the  product.  The  third  iteration  is  the  final  design,  which 
captures  improvements  to  the  design  generated  by  testing,  analysis,  and 
other  forms  of  learning.  These  iterations  can  be  seen  in  figure  10,  which 
provides  a  general  comparison  of  the  amount  of  systems  engineering 
accomplished  in  the  successful  and  problematic  cases  prior  to  launching  a 
product  development  program. 


Figure  10:  Systems  Engineering  Used  for  Successful  and  Problematic  Cases 


Successful  Programs  Used 
Systems  Engineering  Before 
Product  Development 
Began 


In  each  successful  case,  the  product  developer  worked  closely  with  the 
customer  to  understand  its  wants,  which  were  often  articulated  in  a 
bottom-line  metric,  such  as  cost  per  passenger  mile  for  a  commercial 
airplane.  By  the  time  the  developer  had  committed  to  a  product 
development  program,  it  was  well  into  its  systems  engineering  process  and 
had  developed  a  preliminary  design  of  the  product.  This  process  identified 
the  gaps  between  resources  and  expectations,  which  could  be  then 
addressed  through  investments,  alternate  designs,  and,  ultimately, 
trade-offs.  The  knowledge  produced  by  the  process  put  the  developer  in  a 
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good  position  to  negotiate  with  the  customer  because  consequences  could 
be  associated  with  attempts  to  meet  those  wants  that  exceeded  the 
developer’s  capabilities.  The  process  also  involved  the  customer  through 
periodic  reviews  and  acceptance  of  the  product’s  final  design.  According  to 
some  commercial  representatives,  systems  engineering  is  a  good 
investment  to  reduce  risk,  usually  comprising  a  small  percentage  to  the 
overall  development  cost  of  a  new  product.  We  found  two  commercial 
firms — Caterpillar  and  Bombardier  Aerospace — and  two  DOD  programs — 
the  Army’s  Tactical  Unmanned  Aerial  Vehicle  and  the  Air  Force’s  Global 
Hawk  unmanned  aerial  vehicle — that  employed  fairly  extensive  systems 
engineering  before  they  committed  to  product  development. 

Caterpillar’s  797  Mining  Truck  Caterpillar’s  New  Product  Introduction  process  calls  for  completing  a 

significant  amount  of  the  systems  engineering  process  during  the  concept 
phase  for  any  new  product  before  product  development.  It  applied  this 
process  to  its  development  of  the  797  mining  truck,  a  new  product  design. 
It  spent  significant  time  and  effort  prior  to  beginning  product  development 
establishing  the  customers’  wants  and  closing  the  gap  between  them  and 
available  resources.  Before  committing  to  its  development,  Caterpillar 
gathered  information  from  the  mining  companies  about  operating 
conditions  and  the  cost  per  ton  of  ore  hauled  desired  of  the  new  truck  that 
they  would  be  willing  to  accept  during  the  truck’s  lifetime.  Caterpillar  then 
made  a  preliminary  determination  of  what  the  truck’s  performance 
requirements  would  have  to  be  to  meet  these  conditions  and  costs.  For 
example,  it  determined  that  the  truck’s  payload,  or  bed,  would  have  to  haul 
at  least  360  tons,  travel  at  certain  speeds,  and  climb  certain  grades.  This 
information  served  as  the  starting  point  for  systems  engineering  to 
determine  if  the  performance  requirements  for  the  797  were  achievable 
given  Caterpillar’s  resources.  Once  these  requirements  were  established, 
Caterpillar’s  engineers  began  an  iterative  systems  engineering  process  to 
design  a  product  that  would  meet  the  customers’  wants  and  could  be 
developed  within  funding,  schedule,  and  resource  targets.  This  process 
culminated  in  a  specific  product  solution  that  matched  requirements  to 
resources  before  Caterpillar  committed  to  product  development. 

The  systems  engineering  process  for  the  797  forced  key  trade-offs  between 
performance  requirements  and  design  solutions  prior  to  commitment  to 
product  development.  For  example,  to  transmit  the  power  from  the  engine 
to  the  rear  wheels,  the  original  design  called  for  using  very  large  differential 
gears.  Upon  reviewing  that  design,  an  experienced  Caterpillar  production 
engineer  noted  that  no  gear  manufacturer  made  a  gear  that  large  and  that  to 
create  such  a  production  capability  would  be  risky.  Consequently,  the 
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design  engineers  found  an  alternative  that  called  for  making  incremental 
changes  in  the  transmission  and  wheel  designs,  which  enabled  existing 
differential  gears  to  be  used.  These  design  changes  and  the  processes 
required  to  make  the  changes  were  demonstrated  successfully  prior  to 
product  development.  Systems  engineering  also  revealed  risk  on  some  of 
the  new  hydraulic  technology  for  the  797  that  would  allow  easier  and  more 
reliable  unloading.  Because  this  new  technology  had  not  been  used  in  the 
field  before,  Caterpillar  engineers  demanded  demonstrations  and  field  tests 
of  the  technology  before  allowing  it  onto  the  797  truck. 

Bombardier  set  requirements  for  a  new,  larger  family  of  regional  jets  that 
will  carry  up  to  115  passengers — the  BRJ-X  series — using  its  Bombardier 
Engineering  System.  Beginning  with  an  overall  cost-per-passenger-mile 
target  from  the  customer,  Bombardier  employed  an  iterative  systems 
engineering  process  to  make  trade-offs  between  performance 
requirements — which  were  based  on  extensive  market  analysis — and 
available  resources  to  arrive  at  requirements,  design,  and  cost  and  schedule 
targets  before  committing  to  product  development.  The  company  plans  to 
achieve  nearly  full  knowledge  of  the  product’s  design  before  it  commits  to 
product  development.  At  launch,  the  product’s  performance  requirements 
and  its  configuration  will  be  frozen. 

During  Bombardier’s  trade  study  process,  customer  wants  are  thoroughly 
challenged  by  the  firm’s  assessment  of  technology  capability  and  readiness, 
as  well  as  by  manufacturing,  producibility,  logistics,  and  other  implications 
of  the  product’s  features.  During  this  process,  engineers  are  cognizant  that 
product  success  and  profitability  hinge  on  not  only  product  features  but 
also  product  price  and  quantity  that  market  research  has  determined.  This 
fact  tempers  decisions  about  the  viability  of  key  performance  or  design 
parameters.  In  other  words,  if  Bombardier  cannot  produce  and  develop  a 
product  within  a  customer’s  price  range,  then  the  customer  is  not  likely  to 
buy  it.  Once  this  process  is  complete,  Bombardier  performs  a  detailed 
aircraft  review,  which  results  in  a  trade-off  analysis.  If  this  analysis 
suggests  that  key  performance  requirements — speed  or  range,  for 
example — cannot  be  achieved  within  the  established  price  and  quantity,  the 
requirements  are  changed.  If  product  performance  is  reduced  below  the 
customer’s  minimum  threshold,  profit  calculations  are  upset  and  the 
development  program  is  not  initiated. 
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Army’s  Tactical  Unmanned  The  Army’s  Tactical  Unmanned  Aerial  Vehicle  is  a  DOD  program  that  used 

Aerial  Vehicle  systems  engineering  to  gather  knowledge  prior  to  starting  development. 

The  vehicle  must  meet  the  brigade  commanders’  need  for  a  day  or  night, 
adverse  weather,  multisensor  data  collection  system  with  improved 
communication  with  joint  forces  that  provides  real-time  battle  information 
and  cannot  be  observed  by  the  enemy.  The  program  grew  out  of  an 
Advanced  Concept  Technology  Demonstration1  program  that  was  started  in 
1996  and  completed  in  1998.  While  the  air  vehicle  that  was  flown  and 
evaluated  during  this  demonstration  met  most  of  the  Army’s  close-range 
reconnaissance  requirements,  it  did  not  meet  all  requirements  and  was 
canceled.  Figure  11  shows  the  Army’s  Tactical  Unmanned  Aerial  Vehicle. 


1  Advanced  Concept  Technology  Demonstrations  attempt  to  bring  mature  technologies 
together  to  meet  an  existing  military  need.  They  are  managed  by  science  and  technology 
organizations  and  must  demonstrate  military  utility  before  they  can  become  acquisition 
programs. 
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Figure  11:  Tactical  Unmanned  Aerial  Vehicle 


The  Army  used  systems  engineering  to  transform  customer  expectations  into  a  set  of  achievable 
product  requirements  prior  to  program  launch. 


The  technology  demonstration  provided  the  Army  with  knowledge  about 
the  achievability  of  the  customers’  requirements.  The  product  developer 
built  a  prototype  system  that  was  used  to  identify  gaps  between  what  the 
customer  wanted  and  what  was  achievable  with  resources.  Through 
systems  engineering,  the  developer  transformed  the  wants  into  a  set  of 
performance  requirements  that  led  to  the  prototype’s  design.  The  Army 
used  the  results  of  the  technology  demonstration  to  define  the  customer’s 
requirements  that  were  geared  toward  obtaining  a  system  that  required 
minimal  development,  based  largely  on  what  was  demonstrated.  Because 
of  the  knowledge  gained  from  systems  engineering,  the  Army  was  able  to 
stage  a  fly-off  of  four  prototype  designs  within  9  months  after  the  product 
development  program  began.  The  fly-off  results  provided  additional 
knowledge  to  help  match  customer  wants  with  the  product  developer’s 
resources  through  further  trades.  The  Tactical  Unmanned  Aerial  Vehicle 
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that  is  now  in  development  closely  resembles  the  design  that  won  that 
competition.  Another  program,  the  Air  Force’s  Global  Hawk,  followed  a 
similar  path  by  conducting  systems  engineering  and  building  prototype 
systems  prior  to  starting  its  program. 


Problems  Arose  When  Only 
Limited  Systems 
Engineering  Was  Done  Prior 
to  Launch 


In  the  problematic  cases,  product  developers  had  not  progressed  as  far  into 
the  systems  engineering  process  at  the  time  the  program  was  launched.  In 
most  of  these  cases,  the  product  developers  had  only  notional  designs  at 
that  point — not  thorough  enough  to  translate  expectations  into  specific 
functions  against  which  resources  could  be  compared.  It  was  not  until 
product  development  was  underway  that  systems  engineering  was  fully 
employed  to  create  a  preliminary  product  design.  In  each  case,  this 
occurred  several  years  after  the  acquisition  program  was  started.  The  lack 
of  knowledge  at  the  start  of  each  program  made  well-informed  trades 
between  customer  wants  and  developer  resources  difficult  to  see  or  make. 
Nonetheless,  cost  and  schedule  targets  for  each  program  were  set  based  on 
available  information.  Problems  that  were  discovered  during  product 
development  and  during  the  systems  engineering  process  often  resulted  in 
the  need  for  more  time  or  money  than  had  been  estimated  at  program 
launch. 


Two  DOD  programs  we  reviewed — the  Radio  Frequency  Countermeasures 
system  and  the  Crusader  program — initiated  a  systems  engineering  process 
before  product  development;  however,  it  was  not  extensive  and  performed 
by  the  eventual  product  developer  hired  to  design  the  product  and  only 
resulted  in  a  notional  product  design  prior  to  the  decision  to  commit  to 
product  development.  The  product  developers  did  not  gain  significant 
knowledge  about  what  was  possible,  given  the  availability  of  resources, 
until  after  the  decision  was  made  to  commit  the  resources  toward 
developing  the  product. 


Navy’s  Radio  Frequency  The  majority  of  the  systems  engineering  by  the  Radio  Frequency 

Countermeasures  System  Countermeasures  product  developer — the  defense  firm  that  was  awarded 

the  development  contract — was  not  done  until  after  the  Navy  had  launched 
the  program.  The  knowledge  used  to  match  requirements  with  funding, 
schedule,  and  other  resources  was  limited  before  the  program  was 
launched.  The  Navy  did  enough  analysis — about  25  percent  of  systems 
engineering,  according  to  the  former  program  manager — to  form  a  notional 
design  of  the  system.  The  notional  design  was  based  on  a  top-level  analysis 
of  the  functions,  which  was  done  primarily  by  the  Navy’s  program  office. 
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The  Navy  nonetheless  considered  this  amount  of  knowledge  sufficient  to 
commit  to  product  development. 

The  Navy  entered  product  development  assuming  that  a  large  number  of 
the  needed  parts  would  be  nondevelopmental  items — items  already  used 
on  products.  The  Navy  estimated  that  90  percent  of  the  parts  for  one  of  the 
most  critical  subsystems,  the  techniques  generator,2  would  be  mature, 
readily  available  items.  According  to  the  program  manager,  it  was  not  until 
the  contractor  hired  to  develop  the  system  began  the  detailed  systems 
engineering  process  that  it  discovered  that  only  about  50  percent  of  those 
parts  were  still  available.  Many  of  the  required  parts  had  become  obsolete 
and  had  to  be  replaced  with  redesigned  or  newly  developed  parts — 
revealing  a  gap  between  what  was  thought  to  be  within  the  developer’s 
capabilities  and  what  the  requirements  were.  As  a  result,  problems  with  the 
assumptions  made  about  the  notional  design  by  the  Navy  were  discovered 
in  product  development,  after  resources  had  been  determined  and  the 
fielding  date  had  been  established.  This  resulted  in  significant  disruptions 
to  planned  cost  and  schedule;  eventually  affecting  the  product’s  fielding. 

Army’s  Crusader  Artillery  Vehicle  The  Crusader  artillery  vehicle  is  another  case  in  which  systems  engineering 

was  not  performed  by  the  product  developer,  to  a  large  extent,  until  after 
the  acquisition  program  had  been  launched.  The  development  program  was 
launched  based  on  a  notional  design  done  by  the  Army’s  program  office, 
not  extensive  systems  engineering  done  by  the  contractor  that  would 
design  and  build  the  product.  Key  to  this  notional  design  was  the  use  of  a 
liquid  propellant — new  technology — for  firing  weapon  projectiles.  Program 
officials  stated  that  the  optimal  range  that  the  Crusader  is  required  to  fire  a 
weapon  is  still  based  on  the  use  of  this  liquid  propellant.  The  Army 
assessed  various  aspects  of  the  risk  of  developing  the  liquid  propellant 
technology  and  integrating  it  into  the  weapon  system  between  low  and 
moderately  high.  Nevertheless,  on  the  basis  of  the  notional  design,  the 
Army  committed  to  launching  product  development. 

After  the  program  was  launched  in  1994,  the  product  developer  was 
awarded  a  contract  to  develop  the  Crusader.  According  to  a  program 
official,  it  took  2  years  of  systems  engineering  to  determine  if  the 
requirements  were  feasible  given  established  cost  and  schedule  targets.  In 
1996,  the  product  developer  determined  that  the  liquid  propellant 


2  A  jamming  device  carried  onboard  the  aircraft  produces  jamming  signals  that  are 
transmitted  by  fiber  optic  cable  to  the  towed  decoy  for  transmission. 
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technology  was  high  risk  in  all  aspects  and  that  it  would  cost  an  additional 
$500  million  to  develop.  This  was  much  more  than  originally  estimated  at 
the  start  of  the  program  and  represented  a  m^jor  resource  gap.  As  a  result, 
the  Army  elected  to  use  a  more  traditional,  less  capable  back-up — a  solid 
propellant — to  achieve  a  match  between  available  resources  and 
requirements.  This  decision  may  impact  system  performance  since, 
according  to  program  officials,  the  Crusader  will  probably  not  achieve  its 
optimal  firing  range.  The  firing  range  and  the  liquid  propellant  were  part  of 
the  reason  for  launching  the  program  in  the  first  place.  Figure  12  shows  the 
Crusader  artillery  vehicle. 
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Flexibility  in  Setting 
Requirements  Is  Key  to 
Closing  Gaps  Between 
Customer 
Expectations  and 
Developer  Resources 


While  knowledge  is  essential  to  identifying  gaps  between  expectations  and 
resources,  it  takes  flexibility  on  the  part  of  both  the  customer  and  the 
product  developer  to  close  the  gaps.  Flexibility  represents  the  customer’s 
ability  and  willingness  to  lower  product  expectations,  coupled  with  the 
product  developer’s  willingness  and  ability  to  invest  more  resources  to 
reduce  technical  risks  and  other  gaps  before  program  start.  Flexibility, 
when  informed  by  systems  engineering  knowledge  prior  to  program 
launch,  was  essential  to  lowering  risk  and  reducing  the  cost  and  length  of 
product  development  because  knowledge  alone  does  not  produce 
trade-offs.  Absent  such  flexibility,  resources  and  wants  can  still  be  matched 
before  starting  product  development,  but  the  options  to  closing  any  gaps 
exposed  by  systems  engineering  are  limited  to  additional  investments  on 
the  developer’s  part.  The  scenario  with  the  most  potential  for  costly 
problems  is  one  in  which  neither  the  requirements  are  flexible  nor 
sufficient  systems  engineering  has  been  done  to  reveal  resource  gaps 
before  launch.  Several  of  the  DOD  programs  we  reviewed  that  had 
significant  problems  in  meeting  product  development  objectives  followed 
this  scenario. 


While  there  are  instances  in  which  flexibility  on  the  part  of  the  customer  is 
inherent,  such  as  for  a  unique  product  that  has  no  predecessor,  in  most 
cases,  potential  customers  for  a  new  product  already  have  expectations.  In 
these  cases,  flexibility  has  to  be  fostered.  We  found  two  factors  that 
fostered  such  flexibility  before  product  development  started.  First,  product 
development  cycle  times  were  limited.  This  forced  product  developers  and 
customers  to  agree  on  requirements  that  were  achievable  within 
established  time  limits.  Second,  the  developer  enlisted  the  customer’s  trust 
through  an  evolutionary  approach  to  product  development,  which  relieved 
the  customer  of  the  pressure  to  want  all  needs  met  in  a  single  product 
iteration.  These  factors  made  customers  more  willing  to  defer 
requirements  that  demanded  more  time  or  unproven  technologies  for 
succeeding  versions  of  the  product.  In  contrast,  when  these  factors  were 
not  present  and  trade-offs  were  not  made,  there  was  an  implicit  decision  to 
accept  the  risk  of  not  having  the  product  or  the  capability  when  needed. 


Requirements  Were  Flexible 
Until  Program  Launch  on 
Successful  Cases 


In  successful  cases,  requirements  were  flexible  until  the  decision  was  made 
to  commit  to  product  development  because  both  customers  and  developers 
wanted  to  limit  cycle  time.  This  made  it  acceptable  to  reduce,  eliminate,  or 
defer  some  customer  wants  so  that  the  product’s  requirements  could  be 
matched  with  the  resources  available  to  deliver  the  product  within  the 
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desired  cycle  time.  The  customer  had  incentives  to  trade  off  some  wants, 
owing  to  the  desire  to  get  the  product  within  cost  and  time  predictions  and 
to  confidence  that  future  versions  of  the  product  would  meet  many  of  those 
wants.  The  two  commercial  companies  we  visited  exhibited  these 
characteristics.  In  DOD,  we  found  two  programs  that  had  exhibited  this 
flexibility  before  launch. 

Because  of  mutual  interest  on  the  part  of  the  product  developer  and  the 
customer,  Caterpillar’s  board  of  director’s  mandate  for  the  new  mining 
truck  to  be  developed  before  the  end  of  1998  became  a  catalyst  for 
flexibility  in  requirements  and  design  decisions.  For  example,  Caterpillar 
wanted  to  introduce  new  prognostic  technology  on  the  797  that  could 
significantly  lower  the  cost  of  operating  the  truck — an  expressed  want  of 
the  customer.  At  the  heart  of  this  technology  were  monitoring  sensors  that 
could  assist  in  forecasting  wear  and  tear  on  the  truck — such  as  on  the 
powertrain,  brakes,  and  tires — thereby  allowing  better  maintenance 
management  that  would  reduce  how  often  components  had  to  be  replaced 
and  increase  the  truck’s  life.  However,  Caterpillar  engineers  concluded  that 
there  was  not  enough  time  to  make  this  technology  mature  enough  to  be 
included  in  the  initial  design  if  the  18-month  delivery  schedule  was  to  be 
met.  Despite  the  desire  to  have  the  sensors  on  the  797,  the  customer 
understood  the  need  to  make  a  trade-off  and  was  willing  to  do  without  the 
sensor  on  the  initial  version  if  it  meant  getting  the  truck  on  time.  As  a 
result,  Caterpillar  deferred  the  technology,  despite  the  increased 
performance  it  offered,  because  it  would  put  other  program  resources — 
most  importantly,  schedule — at  risk.  The  customer  accepted  this  trade-off. 
The  trade-off  was  made  possible  by  Caterpillar’s  recognition  of  the  risk,  the 
customer’s  trust  in  the  next  generation  of  the  truck,  and  the  mutual  desire 
to  deliver  the  basic  product  on  time  and  within  cost. 

Bombardier’s  goal  is  to  limit  cycle  time  on  all  product  developments  to 
36  months.  At  the  start  of  a  new  product’s  development  process, 
decisionmakers  ask,  “What  can  we  accomplish  within  36  months?” 
Bombardier  representatives  explained  that  this  forces  them  to  have 
significant  knowledge  about  a  new  product’s  cost  and  performance  early  in 
the  process  and  is  consistent  with  an  evolutionary  approach  to  product 
development.  If  a  product’s  requirements  force  them  to  consider  longer 
development  time  frames,  they  must  consider  a  future  generation  of  the 
product  or  a  new  and  different  product.  This  time  frame  played  a  role  in  the 
decision  to  reduce  the  BRJ-X  customer’s  requirement  for  speed.  The  initial 
speed  requirement,  based  on  market  research,  was  originally  .81  mach. 
After  subjecting  the  requirement  to  systems  engineering,  Bombardier’s 
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engineers  deduced  that,  while  speeds  of  up  to  .78  mach  could  be  achieved 
with  current  off-the-shelf  propulsion  technology,  new  and  undemonstrated 
propulsion  technology  would  be  required  to  achieve  .81  mach.  After  gaining 
additional  knowledge  through  wind  tunnel  testing,  the  engineers  concluded 
that  to  design  and  develop  the  needed  propulsion  and  aerodynamics  would 
create  significant  risk  for  the  36-month  product  development  time  frame. 

Before  launching  the  BRJ-X  development  program,  Bombardier’s 
marketing  staff  went  back  to  the  customers — over  30  airlines — and 
presented  the  information  showing  that  the  additional  speed  would  require 
more  development  time  and  thus  be  more  costly.  On  the  strength  of  the 
analysis,  the  customers  agreed  that  .78  mach  would  still  meet  their  needs. 
In  fact,  they  discovered  that  most  European  air  traffic  controllers  would 
not  allow  the  higher  speed.  Based  on  the  agreement,  Bombardier  reduced 
the  requirement  to  .78  mach,  allowing  the  use  of  a  low-risk  propulsion 
system.  The  36-month  goal  forced  the  developer  and  the  customers  to 
negotiate  between  the  product’s  requirements  and  resources,  while 
systems  engineering  provided  the  knowledge  so  that  informed  trade-offs 
could  be  made. 

In  two  DOD  cases  that  have  met  product  development  objectives  so  far  — 
the  Air  Force’s  Global  Hawk  program  and  the  Army’s  Tactical  Unmanned 
Aerial  Vehicle  program — senior  acquisition  executives  placed  resource 
constraints  on  the  programs  and  advocated  evolutionary  acquisitions, 
which  provided  the  customers  and  product  developers  the  incentives  and 
the  opportunity  to  cooperate  and  make  mutually  beneficial  trade-offs.  The 
Tactical  Unmanned  Aerial  Vehicle  had  a  schedule  for  first  delivery  of  a 
system  within  l-Vz  years  after  launch  and  the  Army  established  an 
evolutionary  acquisition  strategy  for  the  product  to  meet  this  schedule. 
This  mandate  was  driven  by  a  mutual  agreement  between  senior  executives 
from  the  Army’s  acquisition  and  requirement  setting  communities  that  the 
priority  is  to  field  it  quickly.  This  agreement  fostered  a  good  relationship 
between  the  two  communities  that  allowed  a  “no  bells  and  whistles” 
approach  to  development. 

The  product  manager  told  us  that  using  the  guidance  as  a  foundation,  the 
customer  agreed  to  collaborate  on  a  basic  list  of  key  performance 
parameters  as  the  only  “must-have”  requirements  for  the  initial  Tactical 
Unmanned  Aerial  Vehicle  and  define  the  remaining  requirements  as 
tradable  against  resources,  such  as  cost  and  schedule.  The  requirements 
that  were  not  key  to  the  initial  vehicle  were  grouped  into  three  categories, 
each  more  important  than  the  next,  but  none  important  enough  to  be  added 
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to  the  initial  vehicle  if  not  achievable  within  the  resources  available  to  the 
developer.  Product  developers  stated  that  several  requirements  would  have 
been  difficult  to  achieve  without  additional  time  and  money  to  develop 
solutions.  When  they  explained  this  to  the  customer  and  pointed  out  the 
associated  high  risks  for  achieving  the  requirements  within  the  time 
frames,  the  customer  was  willing  to  defer  these  requirements  to  future 
versions  of  the  system  in  order  to  meet  the  planned  development  schedule. 

The  requirement  for  the  Tactical  Unmanned  Aerial  Vehicle’s  imagery 
capabilities — that  is,  the  electro-optic  and  infrared  sensor  systems  that  are 
used  for  observing  targets— is  a  good  example  of  this  flexibility.  The 
customer  originally  wanted  the  imagery  to  have  at  least  a  90-percent 
probability  of  recognizing  and  identifying  targets  from  an  altitude  of  6,000 
to  8,000  feet.  On  the  basis  of  a  systems  engineering  analysis,  the  product 
developer  concluded  that  this  was  not  achievable  with  the  technology 
available  within  the  I-V2  year  time  frame  and  that  more  time  and  money 
would  be  required  to  develop  technology  that  would  meet  the  capability.  As 
a  result,  the  customer  reduced  the  requirement  to  70  percent,  which  was 
both  technically  feasible  and  still  useful  to  the  customer,  and  made 
90  percent  a  goal  that  might  be  achieved  in  future  generations. 

A  similar  combination — a  cost  limitation  and  urgent  customer  need — led  to 
flexible  requirements  on  the  Global  Hawk  unmanned  aerial  vehicle.  A 
development  cost  limit  set  by  DOD  executives  forced  the  customer  and  the 
program  office  to  prioritize  requirements  into  what  could  be  achieved 
within  the  limit.  Additionally,  the  customer,  the  United  States  Joint  Forces 
Command,  assessed  the  usefulness  of  the  system  and  stated  the  basic 
capability  tested  during  the  Advanced  Concept  Technology  Demonstration 
could  provide  utility  today.  This  assessment  increased  the  importance  of 
quick  delivery  and  forced  the  requirement  setters  and  the  product 
developer  to  look  at  an  evolutionary  approach  to  meeting  their  needs, 
planning  for  a  basic  capability  in  the  first  product  line  and  more  advanced 
capabilities  added  over  time.  Together,  these  constraints  gave  the  program 
manager  leverage  in  persuading  the  user  to  reduce  or  defer  requirements. 
For  example,  the  requirements  manager  wanted  a  more  advanced  imagery 
system  in  the  first  product  line.  However,  the  program  manager  pointed  out 
to  the  requirements  manager  that  the  cost  and  time  needed  to  include  this 
capability  would  exceed  the  cost  limit.  In  response,  the  requirements 
manager  agreed  to  defer  this  capability  to  a  later  version  of  the  product. 
Figure  13  shows  the  Global  Hawk  unmanned  aerial  vehicle. 
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The  Air  Force  has  adopted  an  evolutionary  acquisition  strategy  to  match  customer  expectations  with 
the  developer’s  resources,  enabling  quicker  delivery  of  the  Global  Hawk. 


Less  Successful  Cases 
Maintained  Inflexible 
Requirements  Prior  to 
Launch 


On  the  weapon  system  programs  that  did  not  meet  development  objectives, 
the  products’  performance  requirements  were  generally  not  flexible  prior 
to  launch — a  condition  compounded  by  the  fact  that  systems  engineering 
had  not  been  done  in  time  to  reveal  gaps  between  requirements  and 
resources.  The  requirements  demanded  advanced  technologies  that  were 
not  yet  proven  or  available.  The  matching  of  requirements  with  available 
resources  was  not  done  until  well  into  the  product  development  program. 
However,  the  requirements  were  not  reduced.  Instead,  additional 
resources,  including  technologies,  time,  and  money,  had  to  be  added.  We 
have  reported  on  the  negative  consequences  of  basing  cost  and  schedule 
estimates  on  the  hoped-for  success  of  such  technologies.3  Without  the 
countervailing  presence  of  systems  engineering  knowledge  from  the 


3 Best  Practices:  Successful  Application  to  Weapon  Acquisitions  Requires  Changes  in  DOD's 
Environment  (GAO/NSIAD-98-56,  Feb.  24,  1998).  Best  Practices:  Better  Management  of 
Technology  Development  Cat 1  Improve  Weapon  System  Outcomes  (GAO/NSIAD-99-162, 
July  30, 1999). 
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product  developer  or  the  customer’s  acceptance  of  an  evolutionary  product 
development  approach,  flexibility  was  difficult  to  engender. 

Army’s  Comanche  Helicopter  In  the  Comanche  program,  initial  requirements  for  attacking  targets  and 

performing  other  tasks,  when  combined  with  the  desire  for  small  size  and 
light  weight,  called  for  a  unique  capability  compared  with  existing 
helicopters.  Meeting  the  size  and  weight  requirements  depended  on  new 
technologies  such  as  advanced  forward  looking  infrared  and  integrated 
avionics.  Both  size  and  weight  were  critical  elements  of  a  mission 
equipment  package  that  was  supposed  to  reduce  the  pilot’s  workload  while 
improving  aircraft  performance  in  the  areas  of  communications,  targeting 
range,  and  pilot  capabilities.  The  program  office  stated  that  at  program 
launch  the  technologies  were  still  conceptual  in  nature,  needing  an 
investment  in  time  and  funding  before  they  would  be  mature  enough  to 
satisfy  requirements.  As  a  result,  Comanche  was  expected  to  provide  a 
quantum  leap  in  product  performance  using  new  technologies  that  were 
not  fully  understood  at  the  time  the  program  began. 

Not  only  did  the  customer  consider  the  requirements  that  made  these 
technologies  necessary  as  not  tradable,  they  were  also  confined  by  weight 
and  cost  restrictions  placed  on  the  program.  The  cumulative  effect  of 
several  competing  requirements — low  cost,  reduced  weight,  long  range, 
and  increased  lethality — resulted  in  solutions  that  relied  on  advanced 
technologies.  For  example,  a  more  mature  target  sensing  technology  that 
might  have  met  performance  requirements  was  rejected  because  it  weighed 
too  much.  The  Army  decided  to  launch  the  program  despite  the  significant 
lack  of  knowledge  about  the  needed  technologies,  leaving  a  mismatch 
between  requirements  and  available  resources,  and  chose  to  develop  the 
new  technologies  during  the  product  development  program.  This  action 
placed  the  burden  of  maturing  technologies  onto  the  program  manager 
during  product  development.  Difficulties  encountered  maturing  these 
technologies  to  meet  unyielding  user  requirements  contributed,  in  part,  to 
the  program’s  significant  cost  and  schedule  increases. 

Army’s  Crusader  Artillery  Vehicle  The  Army’s  Crusader  program  also  had  competing  requirements  that— 

when  taken  together — resulted  in  an  inflexible  system  solution. 
Requirements  for  firing  range,  accuracy,  rate  of  fire,  lethality,  and  resupply 
remained  inflexible  prior  to  program  launch.  Similar  to  the  Comanche 
program,  this  inflexibility  limited  the  technical  solutions  that  could  meet 
the  requirements  and  forced  the  program  to  be  launched  with  immature 
technologies.  Many  of  these  technologies — such  as  the  automated 
ammunition  loading  and  handling  system  and  the  actively  cooled  cannon 
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barrel — were  considered  technological  firsts  for  U.S.  artillery  systems.  As  a 
result,  the  Crusader  program  is  expected  to  take  over  14  years  and  cost 
over  $4  billion  to  develop. 

Army’s  Brilliant  Anti-armor  The  BAT  program  also  had  inflexible  requirements  that  exceeded  available 

Submunition  resources  at  the  start  of  the  program.  The  BAT  operational  concept  is  based 

on  a  need  to  locate  targets  from  great  distances  using  advanced  sensor 
technologies,  which  are  then  required  to  guide  the  weapon  to  closer  ranges, 
where  the  submunition  would  engage  the  target.  The  acoustic  target 
technology  was  the  most  important  technology  needed  to  meet  the 
weapon’s  performance  requirements,  and  the  contractor  proposed  a 
weapon  concept  based  on  it.  The  Army  accepted  the  concept  and  drafted 
performance  requirements  based  on  the  acoustic  technology.  In  this  case, 
there  was  limited  flexibility  to  negotiate  trade-offs  between  requirements 
and  resources  given  the  small  margin  of  error  for  a  munition  to  hit  the 
target.  Consequently,  the  key  to  matching  requirements  with  resources 
depended  on  committing  the  necessary  resources  to  close  gaps  after  the 
program  was  launched. 

At  the  start  of  product  development,  the  cost  and  schedule  targets  for 
BAT’s  development  were  set  without  knowledge  about  the  feasibility  of  the 
performance  requirements.  While  the  requirements  were  based  on  a 
specific  technology,  it  was  not  mature  enough  for  inclusion  onto  a  product. 
In  fact,  the  technology  was  still  being  defined  in  paper  studies.  The  Army 
did  not  prototype  this  technology  until  almost  6  years  after  the  program 
was  launched.  During  its  product  development  program,  the  BAT 
experienced  significant  development  cost  and  schedule  increases.  Program 
officials  attribute  these  increases,  at  least  in  part,  to  unknowns  about  the 
new  technologies.  Because  requirements  remained  inflexible,  the  product 
developer  was  forced  to  add  resources  to  achieve  a  match  between  the 
two. 


Balance  in  the  Roles 
the  Customer  and 
Product  Developer 
Makes  for  Effective 
Trade-offs 


of  With  knowledge  gained  from  systems  engineering  and  flexible 

requirements  as  preconditions,  successful  product  development  programs 
benefited  from  an  environment  in  which  product  developers  and  customers 
had  balanced  roles  and  shared  responsibilities  for  setting  product 
requirements.  In  two  successful  cases,  product  managers  were  responsible 
for  both  (1)  translating  customer  wants  into  product  requirements  and 
(2)  developing  and  delivering  the  products  within  available  resources.  This 
dual  responsibility  allowed  them  to  make  reasonable  trade-offs  between 


Page  50 


GAO-Ol-288  Best  Practices 


Chapter  3 

Several  Factors  Enable  Customer  Wants  and 
Developer  Resources  to  Be  Matched  Before 
Program  Launch 


performance  requirements  and  resources  prior  to  committing  to  product 
development.  In  two  other  successful  cases,  requirement  setters  and 
product  developers  had  equal  roles  in  establishing  and  changing 
requirements,  using  knowledge  from  systems  engineering  as  a  guide.  In 
unsuccessful  cases,  there  was  no  parity  in  decision-making  between  the 
requirement  setter  and  the  product  developer;  the  requirement  setter  held 
the  upper  hand  by  controlling  the  requirements  without  being  bound  by 
cost  and  schedule  considerations  or  the  developer’s  resources. 


Successful  Programs 
Benefited  From  Parity  in 
Decision-making  Between 
Customer  and  Developer 


In  the  successful  commercial  cases  we  examined,  the  product  developer 
had  ultimate  responsibility  for  defining  the  product’s  requirements  but 
worked  intimately  with  the  customers  to  understand  their  needs.  The  firms 
established  an  environment  that  teamed  customer  representatives  with 
product  development  engineers  in  setting  mutually  agreeable  product 
requirements.  Representatives  from  one  commercial  firm  told  us  that  they 
feel  responsible  for  understanding  a  customer’s  operations  better  than  the 
customer  does.  They  achieve  this  through  their  marketing  people.  Once  the 
marketing  staff  understands  a  customer’s  wants,  they  can  work  with  the 
design  and  production  engineers  to  develop  a  product  solution  that  will 
meet  the  customer’s  needs  and  can  be  met  with  company  resources.  The 
product  manager  makes  final  decisions  on  the  product’s  requirements,  but 
is  mindful  of  the  fact  that  a  dissatisfied  customer  will  not  buy  the  product. 


Caterpillar’s  797  Mining  Truck  An  example  of  how  this  parity  in  decision-making  worked  is  the  Caterpillar 

797  mining  truck  engine.  To  meet  the  customers’  needs,  the  engine  had  to 
produce  at  least  3,400  horsepower.  For  maintenance  and  serviceability 
reasons,  some  mining  companies  wanted  a  single  engine  to  produce  this 
power,  which  Caterpillar’s  marketing  staff  translated  into  a  requirement. 
However,  Caterpillar  did  not  have  such  an  engine  and  its  systems 
engineering  analysis  showed  that  a  new  single-engine  design  could  not  be 
matured  and  demonstrated  within  the  18-month  cycle  time  limit.  Thus, 
product  developers  proposed  a  twin-engine  design  that  met  the 
horsepower  requirement  but  used  an  existing  engine.  Caterpillar’s 
customer  representatives  presented  this  proposal  to  the  customers,  as  well 
as  their  position  that  the  firm  could  either  deliver  the  truck  powered  by 
twin  engines  in  18  months  or  take  longer  to  deliver  a  single  engine  truck — 
but  not  both.  Most  customers  accepted  the  proposal  and  the  position. 


Bombardier’s  New  Regional  Jet  Similar  parity  in  decision-making  was  evident  concerning  Bombardier’s 

fly-by-wire  technology.  The  airlines  wanted  fly-by-wire  on  the  BRJ-X 
aircraft  because  of  the  benefits.  The  benefits,  which  Bombardier’s  systems 
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engineering  confirmed  included  96  percent  fewer  parts  and  1,250  fewer 
pounds  than  hydromechanical  controls.  These  benefits  would  reduce 
engine  power  requirements,  saving  about  3  percent  of  fuel  capacity  and 
reducing  the  need  for  maintenance  significantly.  All  of  these  savings 
responded  directly  to  the  airlines’  performance  needs.  However,  because 
this  technology  had  never  been  integrated  onto  a  regional  jet  before, 
Bombardier’s  product  development  engineers  believed  it  to  be  risky.  Also, 
since  the  workforce  of  regional  jet  pilots  had  never  used  fly-by-wire,  they 
would  have  to  be  trained  on  how  to  use  it,  and  system  engineers  who  had 
the  authority  to  accept  or  reject  the  technology  would  not  commit  to 
including  the  technology  unless  it  was  demonstrated  first.  Bombardier 
closed  the  technology  gap  by  demonstrating  it  on  an  existing  aircraft.  At 
that  point,  Bombardier  was  confident  in  including  fly-by-wire  in  the 
product  requirements  for  the  BRJ-X. 


Less  Successful  Cases  Did 
Not  Have  Parity  Between 
the  Requirement  Setter  and 
the  Product  Developer 


In  two  less  successful  cases,  the  customer  set  the  product  requirements 
with  comparatively  little  input  from  the  product  developer.  These  programs 
were  initiated  with  cost  and  schedule  estimates  that  were,  for  the  most 
part,  based  on  the  customer’s  requirements  and  a  notional  design.  The 
product  developer  was  not  an  equal  partner  in  setting  product 
requirements  and  had  to  launch  programs  without  requisite  resources. 
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Before  product  development  began,  the  program  manager  for  the  Radio 
Frequency  Countermeasures  system  had  been  working  with  the  F/A-18  E/F 
customers  to  match  its  needs  with  available  resources.  The  program 
manager  told  us  that  both  parties  believed  they  had  a  good  match  because 
the  technology  was  mature.  Also,  both  parties  agreed  that  an 
approximately  5JA  year  development  cycle  would  deliver  the  desired 
product.  However,  prior  to  program  launch,  as  part  of  the  normal  process 
used  for  approving  product  requirements,  the  proposed  Navy  requirements 
were  distributed  to  the  other  services.  The  Air  Force  became  interested  in 
the  program  and  requested  that  its  needs  for  the  F-15  and  B-l  be  combined 
with  the  Navy’s  to  develop  a  common  system.  The  Air  Force’s  requirements 
were  much  more  demanding  than  the  Navy’s  because  more  power  was 
needed  to  generate  the  electronic  countermeasures4  for  the  F-15,  a  less 
stealthy  aircraft  than  the  F/A-18  E/F.  These  additional  needs  created  a 
technology  gap  that  required  either  more  time  or  trade-offs. 

The  product  developer  analyzed  the  technology  gaps  and  determined  that, 
given  projected  time  and  money,  the  effort  required  to  meet  them  would 
take  close  to  7  years.  The  Navy  customer  would  not  agree  to  the  additional 
time  and  insisted  that  the  product  developer  deliver  a  product  to  meet  the 
expanded  needs  within  the  original  h-Vz  year  time  frame.  The  customers,  in 
sole  control  of  requirements  setting,  thus  firmed  the  product  requirements. 
The  program  manager  informed  us  that  he  had  little  decision-making 
authority  in  the  matter  and  was  compelled  to  accept  the  customers’ 
schedule.  The  result  was  an  imbalance  between  product  requirements  and 
resources,  which  also  resulted  in  cost  and  schedule  increases.  Because  of 
the  delays  in  developing  the  system,  the  Navy  plans  to  incorporate  a  less 
effective  substitute  onto  the  F/A-18  until  the  product  is  ready.  These 
aircraft  will  have  to  be  retrofitted  with  the  system  when  it  becomes 
available — a  costly  and  inefficient  process.  Ultimately,  the  trade-offs  that 
were  unpalatable  before  launch  became  unavoidable  after  launch. 

Parity  between  the  requirement  setter  and  the  product  developer  did  not 
exist  on  the  Comanche  program.  As  discussed  previously,  the  Comanche 
product  development  program  included  immature  avionics  technologies 


4  Electronic  countermeasures  involve  the  jamming  and  deceiving  of  the  enemy’s  electronic 
devices,  such  as  a  radar  or  communications  system,  by  sending  an  electronic  signal  that 
reduces  or  prevents  their  usefulness.  Because  of  the  more  traditional  design  of  the  F-15,  it 
required  the  Radio  Frequency  Countermeasures  system  to  produce  as  much  as  2  to  3  times 
more  amplified  power  than  that  needed  for  the  F/A-18. 
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that  were  needed  to  meet  demanding  and  inflexible  performance 
requirements.  These  technologies  represented  significant  gaps  between 
what  the  customer  wanted  and  what  the  program  office  could  confidently 
deliver  given  the  resources  available.  Prompted  by  concerns  about  the  time 
needed  to  advance  the  necessary  technologies,  the  program  manager 
proposed  trading  off  some  requirements.  However,  requirements  managers 
informed  us  they  were  unwilling  to  accept  trade-offs  and  that  they  believed 
the  program  manager  was  too  risk  averse.  Consequently,  the  program  was 
launched  with  the  requirements  intact,  despite  the  program  manager’s 
concerns,  and,  cost  and  schedule  estimates  for  its  development  have 
doubled. 
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Within  DOD’s  traditional  acquisition  process,  it  is  difficult  to  employ 
systems  engineering  in  making  trade-offs,  maintain  flexibility  in  customer 
expectations,  and  put  the  product  developers  in  a  balanced  position 
relative  to  the  customers.  As  a  result,  customer  needs — however 
legitimate — Eire  translated  into  product  requirements  that  make 
unreasonable  demands  on  available  resources.  This  pattern  is  perpetuated 
because  resources  are  generally  added  later  during  product  development  to 
close  any  gaps  between  requirements  and  resources.  The  mechanics  of 
obtaining  funding  and  getting  approval  to  start  an  acquisition  program 
dictate  that  events  proceed  in  the  following  sequence:  set  requirements, 
obtain  funding,  launch  program,  and  perform  systems  engineering.  This 
sequence  places  the  knowledge  that  is  needed  to  identify  resource  gaps— 
and  the  power  that  it  gives  to  the  product  developer — at  a  disadvantage  to 
shape  requirements  and  improve  program  outcomes.  Other  aspects  of  the 
process  reward  behaviors  that  can  put  pressure  on  requirements,  making 
them  less  flexible  and  more  difficult  to  meet.  For  example,  the  services 
must  clearly  differentiate  a  new  weapon’s  performance  characteristics 
from  alternatives  to  successfully  compete  for  funding,  encouraging 
detailed  cost  and  schedule  estimates  to  be  made  with  little  knowledge  from 
systems  engineering.  The  weapon  system  programs  that  did  match 
requirements  and  resources  before  launch — the  Tactical  Unmanned  Aerial 
Vehicle  and  the  Global  Hawk — represented  departures  from  this  process 
that  benefited  greatly  from  the  intervention  and  protection  of  top-level 
individuals. 

DOD  has  recently  changed  the  requirements  setting  and  acquisition 
processes  to  better  reflect  best  commercial  practices.  Specifically,  DOD 
has  revised  its  policy  for  approving  new  development  programs  by 
providing  more  latitude  for  technology  maturation  and  other 
knowledge-generating  activities  to  take  place  before  a  program  is 
launched.  The  acquisition  policy  also  supports  the  evolutionary 
development  approach  as  the  preferred  method  of  product  development.  In 
addition,  DOD  has  revised  its  policy  for  developing  requirements,  calling 
for  a  staged  or  stepped  approach  that  will  dovetail  with  evolutionary 
acquisitions.  By  itself,  however,  the  policy  can  do  little  to  foster  the  early 
matching  of  customer  wants  with  developer  resources  because  the 
mechanics  of  obtaining  funding  and  the  pressures  on  requirements  setters 
are  unchanged.  The  incentives  of  the  process,  unlike  policy,  are  more  likely 
to  be  seen  in  decisions  made  on  the  funding  and  approval  of  individual 
programs. 
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Current  Process  Puts 
Requirements  Setting 
and  Systems 
Engineering  on 
Opposite  Sides  of  the 
Launch  Decision 


DOD’s  acquisition  process  makes  it  difficult  to  know  what  resources  will  be 
needed  to  meet  requirements  before  launching  a  program.  Within  this 
process,  systems  engineering  does  not  take  place  until  a  program  has  been 
launched  and  cost  and  schedule  targets  have  been  set.  One  reason  for  this 
is  that  DOD  does  not  typically  sign  contracts  with  product  developers,  who 
are  responsible  for  systems  engineering,  until  after  the  requirements  have 
been  written.  Once  an  acquisition  program  is  approved  and  product 
development  is  funded,  a  product  developer  can  begin  systems  engineering 
and  gaps  between  requirements  and  resources  can  be  identified.  However, 
this  process  does  not  yield  knowledge  about  the  requirements’ 
achievability  until  well  into  product  development.  Even  when  requirements 
are  in  draft  at  the  time  of  launch,  they  are  hard  to  change  because  user 
representatives  have  already  invested  significant  effort  in  preparing  them. 
This  sequence  of  events — setting  requirements,  launching  an  acquisition 
program,  contracting  with  a  product  developer,  and  conducting  systems 
engineering — narrows  the  options  for  closing  gaps  primarily  to  increasing 
cost  and  schedule  estimates. 


DOD’s  process  for  setting  requirements  can  begin  many  years  before  a 
program  is  launched,  and  it  can  be  several  years  after  the  program  is 
launched  before  systems  engineering  knowledge  is  obtained.  The  time 
period  from  the  start  of  the  definition  of  the  user’s  needs  to  the  arrival  of 
systems  engineering  knowledge  that  often  uncovers  gaps  can  approach 
10  years  in  some  instances.  More  than  30  organizations  within  the 
requirements  community  may  have  a  hand  in  determining  a  weapon 
system’s  performance  requirements  before  a  contractor  with  systems 
engineering  expertise  can  identify  the  gaps  between  requirements  and 
available  resources.  This  process  means  the  “doability”  of  the  requirements 
is  often  not  known  with  certainty  until  well  into  product  development  or 
until  a  significant  percentage  of  funds  planned  to  develop  the  system  has 
been  invested.  By  this  point  in  time,  customers’  expectations  have  been  set, 
making  it  difficult  to  change  requirements  if  gaps  between  requirements 
and  available  resources  are  found. 


The  process  used  to  set  requirements  and  begin  development  on  the  Army’s 
Crusader  program  provides  an  example  that  illustrates  the  mechanics  of 
the  process  (see  fig.  14). 
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Figure  14:  Timeline  for  Developing  Crusader’s  Requirements 
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The  user’s  representative  for  the  Crusader  (the  Army’s  Training  and 
Doctrine  Command  artillery  school  at  Fort  Sill,  Oklahoma)  began  drafting 
the  performance  requirements  for  the  Crusader  in  1990.  The  process  to 
define  the  requirements  took  approximately  4  years.  During  this  time,  the 
user  representative  framed  the  needed  features  of  the  Crusader  and 
conducted  exhaustive  analyses  and  trade  studies  to  identify  the  optimal 
point  to  set  a  specific  requirement.  Also,  as  required  by  DOD  acquisition 
policy,  the  user  representative  analyzed  a  notional  Crusader  that  met  these 
requirements  against  other  alternatives,  such  as  improved  versions  of 
existing  artillery  systems.  The  result  of  these  analyses  was  a  set  of 
requirements  from  the  user  representative  and  notional  design  prepared  by 
the  Army  program  office.  In  1994,  the  requirements  for  the  Crusader  system 
were  approved,  the  acquisition  program  was  launched  and  funded,  and 
program  cost  and  schedule  targets  were  baselined. 

By  this  time,  key  features  of  the  Crusader  were  defined.  For  example,  the 
Crusader  would  have  to  use  liquid  propellant — a  revolutionary 
technology— to  propel  the  projectiles  far  enough  to  meet  the  optimal  range 
requirements.  Also,  because  the  Crusader  was  required  to  stay  on  the  firing 
line  during  rearming  and  refueling,  it  would  have  to  have  an  armored  and 
automated  resupply  vehicle  to  keep  the  resupply  crews  protected. 

After  the  program  was  launched,  the  Army  entered  into  a  contract  with 
United  Defense  Limited  Partnership  to  develop  the  Crusader.  Over  the  next 
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4  years,  the  firm  applied  systems  engineering  practices  to  the  requirements 
in  order  to  develop  a  preliminary  design.  During  this  process,  the  firm  had 
to  make  trade-offs  in  order  to  develop  a  design  that  could  stay  within 
resources.  By  1996 — 2  years  into  product  development  and  6  years  since 
the  need  for  the  Crusader  was  identified  and  determined — United  Defense 
Limited  Partnership  concluded  that  resources  needed  to  develop  a  liquid 
propellant  technology  were  not  available.  The  firm’s  only  option  was  to 
accept  a  less  capable  technology — a  conventional  solid  propellant, 
increasing  the  risk  that  the  Crusader’s  firing  range  requirement  may  not  be 
achieved.  According  to  Crusader  officials,  United  Defense  Limited 
Partnership  did  not  develop  a  good  preliminary  design  of  the  system  until 
1998  or  about  8  years  after  the  user  representative  began  the  requirements 
setting  process. 


Incentives  of  Current 
Process  Create 
Pressure  on  Product 
Requirements 


Several  factors  in  the  DOD  environment  create  incentives  to  set 
requirements  high  and  to  resist  trade-offs.  The  competitive  nature  of  the 
process  to  justify  a  new  program  puts  great  pressure  for  a  potential 
weapon’s  requirements  to  stand  out.  Unlike  the  commercial  environment, 
the  user  representatives  are  separate  from  both  the  customer  and  the 
product  developer,  and  thus  play  a  unique  role  that  has  different  interests. 
Once  user  representatives  get  a  set  of  requirements  through  the  very 
difficult  and  lengthy  process  of  approval  before  a  program  can  be 
launched,  they  are  understandably  reluctant  to  change  them.  Finally,  the 
DOD  customer — unlike  its  commercial  counterpart — is  more  tolerant  of 
schedule  delays  and  cost  increases. 


Pressures  on  the 

Requirements-setting 

Process 


The  competition  within  DOD  to  win  funding  and  get  approval  to  start  a  new 
program  is  intense;  establishing  the  basic  need  and  writing  requirements  is 
perhaps  the  most  important  step  in  the  process.  This  creates  strong 
incentives  for  requirements  setters  to  write  performance  requirements  that 
will  make  their  particular  weapon  system  stand  out  from  existing  or 
alternative  systems.  Much  of  the  requirement-setting  activity  prior  to 
initiating  a  program  is  devoted  to  proving  the  superior  cost-effectiveness  of 
the  preferred  system  over  others,  with  less  consideration  given  to  the 
resources  that  will  be  needed  to  develop  the  system.  If  user  representatives 
do  not  write  requirements  that  will  withstand  scrutiny  and  prevail  over 
alternatives,  the  program  could  be  killed  and  no  capability  is  gained  for  the 
customer.  Costly,  labor-intensive  studies,  such  as  analyses  of  alternatives, 
are  done  to  determine  the  best  possible  solution  to  meet  the  user’s  needs 
and  to  beat  the  competition.  At  the  same  time,  overall  DOD  funding 
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constraints  put  a  high  priority  on  affordability,  making  it  important  for 
program  sponsors  to  provide  cost  estimates  that  will  fit  within  the  funding 
constraints.  Instead  of  forcing  trade-offs,  challenging  performance 
requirements,  when  coupled  with  other  constraints,  such  as  cost  or  light 
weight,  can  drive  product  developers  to  pursue  exotic  solutions  and 
technologies  that,  in  theory,  could  do  it  all. 

To  simplify  requirements  and  foster  trade-offs,  DOD  has  adopted  the 
practice  of  identifying  only  a  limited  number  of  key  performance 
parameters — “must-haves”  for  the  customer — for  each  weapon  system. 
This  practice  may  help  to  prioritize  requirements.  However,  if  the 
parameters  are  not  informed  by  systems  engineering,  given  the  pressures 
inherent  in  the  process,  they  still  result  in  unrealistic  demands  made  of 
existing  resources.  For  example,  the  Army’s  Crusader  program  had  only 
five  key  performance  parameters,  including  characteristics  such  as 
rate-of-fire,  range,  and  speed,  but  they  were  dependent  on  about  500  other 
subordinate  performance  requirements  that  were  defined  before  systems 
engineering  was  done  by  the  product  developer. 

The  unique  role  of  the  requirement  setter  in  DOD’s  acquisition  process  can 
also  put  pressure  on  requirements.  DOD’s  requirement  setters  are  outside 
of  the  acquisition  community  and  often  represent  an  operational  function 
such  as  air  combat,  artillery,  or  armor.  They  are  not  the  actual  user  like  an 
operating  air  wing  or  Army  brigade,  but  serve  as  a  representative  of  the 
user.  Unlike  the  commercial  world,  where  a  customer’s  wants  are 
represented  by  someone  within  the  product  developer’s  organization, 
DOD’s  requirement  setters  are,  for  the  most  part,  separate  and  independent 
of  the  customer,  program  manager,  and  the  product  developer.  A  user 
representative  can  work  directly  with  DOD  science  and  technology 
organizations  and  defense  firms  to  obtain  information  about  new 
technology,  and  thus  may  be  less  willing  to  defer  to  a  program  manager’s 
advice  on  these  issues.  Also,  while  a  customer  is  focused  on  current 
problems  and  near-term  performance,  a  user  representative  tries  to  look  at 
longer  term  needs.  In  fact,  the  fear  that  DOD  may  not  procure  another  such 
system  for  several  years  creates  incentives  for  user  representatives  to 
reach  for  the  most  capability  possible  because  they  do  not  know  when  or  if 
they  will  get  another  opportunity  to  develop  and  acquire  the  next  weapon 
system. 

The  Comanche  helicopter  program  is  a  good  example  of  these  pressures  at 
work.  The  Comanche  was  initiated  in  the  early  1980s  as  a  family  of 
lightweight,  multipurpose  helicopters  whose  operation  and  support  costs 
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would  be  50  percent  less  than  the  Vietnam-era  fleet.  The  program  was 
originally  expected  to  offer  as  good  a  technical  performance  as  possible 
within  clearly  stated — and  low — unit  cost  goals.  The  requirements  were 
simple  at  the  start.  However,  once  the  requirement  setting  process  began, 
the  program  emerged  as  it  is  today — a  threat-based  program  to  yield  the 
next-generation,  high-performance  helicopter — at  a  cost  significantly 
higher  than  the  existing  Apache.  It  was  justified  as  being  faster,  stealthier 
and  smaller  than  the  Apache  helicopter.  Resource  gaps  were  identified 
after  the  program  was  launched,  when  the  product  developer  began  to 
obtain  knowledge  through  systems  engineering  and  technology 
development  about  what  it  would  take  to  meet  the  requirements. 

Leadership  in  the  Army  has  called  for  a  transformation  in  Army  forces  and 
equipment  to  a  lighter,  more  mobile  force  that  can  be  deployed  more 
quickly  and  easily.  The  impact  this  change  is  having  on  Crusader’s 
requirements  further  illustrates  how  DOD  incentives  can  drive 
requirements  and  ultimately  impact  program  outcomes.  The  preliminary 
Crusader  design  — which  met  all  key  requirements — was  considered  too 
heavy  for  the  new,  lighter  force.  In  fact,  the  Army  Chief  of  Staff  said  he 
would  like  to  have  the  Crusader’s  weight  reduced  to  the  point  that  twice  as 
many  vehicles  could  be  moved  with  the  same  amount  of  transport  aircraft. 
With  the  program’s  future  at  stake,  the  user  representative,  program 
manager,  and  product  developer  are  working  together  to  reduce  the  overall 
design  weight  of  the  system  from  55  to  38  tons.  Because  of  this  new 
priority,  previously  untraded  requirements — such  as  degree  of  crew 
protection,  time  on  the  firing  line,  and  the  need  for  tracked  vehicles — are 
now  being  examined  to  see  if  they  can  be  reduced  to  save  weight. 

A  specific  Crusader  requirement  is  illustrative.  In  developing  the  original 
set  of  requirements,  the  user  representatives  determined  that  the 
Crusader’s  effectiveness  would  be  maximized  if  the  Crusader  could  avoid 
having  to  leave  its  battle  station  to  reload  and  refuel.  To  meet  this 
requirement,  the  resupply  vehicle  would  have  to  reload  and  refuel  the 
howitzer  at  the  battle  station.  This  requirement  necessitated  that  the 
resupply  vehicle  be  armored,  tracked,  and  fully  automated  so  that  the  crew 
would  not  be  exposed  to  enemy  fire.  This  not  only  made  for  a  more 
effective  howitzer  but  also  distinguished  the  Crusader  from  other 
candidates.  In  looking  for  weight  reductions,  the  Army  is  considering 
pulling  at  least  some  of  the  howitzers  off  station  to  reload  and  refuel.  If  it 
relaxes  that  requirement,  the  resupply  vehicle  and  crew  can  do  its  work 
under  safer  conditions.  In  turn,  the  vehicle  would  not  have  to  be  as  heavily 
armored,  automated,  or  tracked.  In  fact,  the  Army  is  considering  using 
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existing  trucks,  which  are  far  lighter  and  less  expensive  than  a  new 
resupply  vehicle.  In  our  view,  the  changed  circumstances  of  the  Crusader 
program — past  the  approval  gate,  a  product  developer  deep  into  systems 
engineering,  and  a  top  level  mandate  to  reduce  weight — created  incentives 
to  make  requirements  trade-offs  that  were  not  acceptable  before. 


Calcification  and  Customer 
Acceptance  Do  Not 
Encourage  Trade-offs 


Once  established,  system  requirements  undergo  incredible  scrutiny  and 
review  by  a  myriad  of  interests  within  DOD’s  process.  For  example,  the 
Crusader  draft  requirements  were  circulated  to  approximately 
30  organizations  throughout  the  Army,  other  services,  operational 
commands,  and  the  development  community  for  review.  This  process 
yielded  943  comments.  To  incorporate  or  otherwise  dispose  of  each 
comment,  a  joint  working  group  with  representatives  from  all  of  the 
reviewing  organizations  was  established.  The  group  incorporated  702  of 
the  943  comments  into  the  requirements,  the  cumulative  effect  of  which 
was  to  add,  rather  than  to  trade,  requirements.  As  one  official  stated  “it  is 
generally  not  the  practice  to  reduce  or  eliminate  requirements  but  to  add 
more  to  appease  a  particular  party.”  This  lengthy  and  cumbersome  review 
process  tends  to  calcify  weapon  system  requirements  before  product 
development  begins  and  knowledge  from  systems  engineering  can  be 
obtained. 


The  practice  of  breaching  cost  and  schedule  objectives  to  meet  difficult 
requirements  would  not  persist  without  a  customer’s  cooperation.  Unlike 
commercial  customers,  DOD  customers  tend  to  be  tolerant  of  cost 
overruns  and  delays  in  order  to  get  a  high-performance  weapon  system. 
Traditionally,  customers  have  been  willing  to  wait  long  periods  of  time  for  a 
highly  desirable  system  that  they  feel  will  provide  them  the  longest  lasting 
capability.  They  would  rather  wait  for  the  most  desirable  system  to  be 
developed  than  accept  a  less  capable  system,  thinking  that  they  may  not  get 
the  opportunity  to  acquire  a  new  or  modified  system  in  the  future.  Again, 
the  Comanche  program  provides  insight.  At  the  time  the  program  was 
started,  the  Army  expected  to  receive  the  first  operational  helicopter  in 
1996.  The  development  program  has  encountered  delays;  some  related  to 
weight,  cost,  and  performance  requirements  that  demanded  immature 
technologies.  However,  the  Army  has  chosen  to  keep  the  requirements 
generally  intact  and  to  do  without  the  helicopter  instead  of  fielding  a  less 
capable  system  more  quickly.  The  Army  now  expects  to  have  an  initial 
operational  capability  in  2006, 10  years  after  the  initial  target  date. 
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More  Successful 
Weapon  System 
Programs  Have 
Departed  From  the 
Normal  Process 


The  two  DOD  programs  we  examined — Tactical  Unmanned  Aerial  Vehicle 
and  Global  Hawk — that  set  requirements  in  a  way  that  approximates  best 
commercial  practices  were  departures  from  DOD’s  normal  process. 
Specifically,  (1)  their  origins  as  Advanced  Concept  Technology 
Demonstrations  enabled  them  to  hire  a  product  developer  much  earlier 
than  is  traditionally  the  case  and  (2)  they  benefited  greatly  from  the 
personal  intervention  of  service  and  DOD  executives  who  created  and 
enforced  conditions  that  were  conducive  to  making  trade-offs  between 
wants  and  resources.  Other  new  programs,  such  as  the  Joint  Strike  Fighter 
program,  showed  that  traditional  incentives  to  speed  a  program  along  still 
existed  and  could  increase  risk  in  product  development. 

The  Advanced  Concept  Technology  Demonstrations,  for  the  Tactical 
Unmanned  Aerial  Vehicle  and  the  Global  Hawk,  showed  the  feasibility  of 
developing  a  system  that  could  do  what  the  customer  wanted  before  a 
commitment  to  product  development  was  made.  The  sequence  of  events 
leading  to  program  launch  on  each  of  these  programs  was  similar  to  those 
of  successful  commercial  firms.  The  demonstrations  involved  the 
customer,  user  representative,  program  manager,  and  product  developer  in 
a  more  integrated  fashion  than  is  normally  the  case.  Moreover,  by  building 
prototypes  for  the  demonstrations,  the  product  developers  had  to  conduct 
a  significant  amount  of  systems  engineering  before  the  programs  were 
launched.  Consequently,  the  demonstrations  provided  valuable  knowledge 
that  allowed  the  customer,  user  representative,  and  product  manager  to 
define  a  set  of  product  requirements  that  could  be  met  within  resources. 
Most  importantly,  the  demonstrations  were  conducted  before  program 
launch,  before  cost  and  schedule  targets  were  set. 

Both  programs  had  unusual  intervention  by  top-level  individuals  that  set 
resource  constraints  and  encouraged  evolutionary  acquisition  strategies. 
On  the  Tactical  Unmanned  Aerial  Vehicle  program,  the  top  military 
acquisition  executive  personally  met  with  the  head  of  the  user 
representative’s  organization  and  struck  an  agreement  that  the  product  was 
to  be  fielded  in  stages,  with  the  first  stage  being  a  very  basic  system.  This 
agreement  had  the  effect  of  establishing  mutual  interest  in  the  same 
program  outcome  and  shielded  the  program  from  criticism  by  either 
community.  The  personal  involvement  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  helped  set  the  stage  for  Global 
Hawk’s  evolutionary  approach  to  meeting  requirements.  The  Under 
Secretary  insisted  on  an  initial  capability  that  could  be  developed  within  a 
fixed  budget  while  providing  the  flexibility  to  defer  other  requirements  to 
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succeeding  versions.  In  both  cases,  this  top-level  intervention  allowed 
requirements  to  be  flexible  and  gave  the  product  developers  parity  with  the 
requirements  setters  in  influencing  requirements.  Equally  important,  we 
believe  the  intervention  signaled  support  for  the  programs,  which  eased 
some  of  the  pressures  that  normally  accompany  efforts  to  get  programs 
approved.  In  each  case,  while  there  was  a  potential  for  success,  it  came  not 
so  much  from  a  well-established  process  but  from  exceptional  behavior 
from  senior-level  leaders.  If,  for  some  reason,  leadership  or  priorities 
change,  the  process  may  not  ensure  success. 


Constructive  Changes 
in  DOD’s  Policy  Not 
Enough  to  Match 
Customer 
Expectations  With 
Developers’  Resources 


While  DOD  has  taken  steps  that  could  help  it  more  efficiently  acquire  its 
weapons,  these  steps  have  not  substantively  changed  the  mechanics  or 
incentives  of  the  requirements  setting  process.  While  the  revised 
acquisition  policy  provides  opportunities  for  improvement,  it  is  not  specific 
about  when  to  match  wants  and  resources.  In  fact,  the  sequence  of 
events — setting  requirements,  obtaining  funding,  launching  an  acquisition 
program,  and  conducting  systems  engineering — remains  essentially  the 
same  as  before  the  revision.  Recent  experiences  with  the  Global  Hawk  and 
the  Joint  Strike  Fighter  programs  indicate  that  traditional  pressures  on  the 
requirements  process  are  still  strong. 


Recent  DOD  Policy  DOD  has  recently  revised  its  policies  for  operation  of  the  defense 

Revisions  Are  Supportive  of  acquisition  and  requirements  setting  processes.  The  acquisition  policy 
Best  Practices  reflects  best  commercial  practices  and  emphasizes  better  use  of 

evolutionary  acquisitions,  more  reliance  on  mature  technology,  and 
reduced  cycle  times  and  costs.  It  recognizes  shorter  acquisition  cycle  times 
as  critical  to  making  the  best  use  of  advanced  technologies  and 
evolutionary  requirements  as  a  way  to  reduce  cycle  times.  The  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  issued 
related  guidance  stating  that  DOD’s  objective  will  be  to  achieve  acquisition 
cycle  times— from  program  start  to  initial  fielding — no  longer  than  5  to  7 
years.  The  revised  acquisition  policy  also  states  a  clear  preference  for 
evolutionary  acquisitions  over  “single  step  to  full  capability”  acquisitions. 
DOD  also  revised  its  acquisition  process,  which  is  now  four  phases: 

(1)  concept  and  technology  development,  (2)  system  development  and 
demonstration,  (3)  production  and  deployment,  and  (4)  operations  and 
support  (see  fig.  15).  In  addition,  guidance  that  directs  the  requirements 
setting  process  was  revised  to  include  an  emphasis  on  the  use  of 
time-phased  performance  requirements  in  support  of  evolutionary 
acquisitions. 
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Figure  15:  Revised  POD  Acquisition  Process _ 
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The  decision  to  launch  a  program  would  normally  take  place  between  the 
first  and  second  phases.  The  revised  policy  states  that  the  decision  is 
dependent  on  three  factors:  technology  maturity,  validated  requirements, 
and  funding.  Assuming  that  technology  is  found  to  be  mature,  at  the  start  of 
the  system  demonstration  and  development  phase,  the  policy  requires  the 
weapon’s  requirements  to  be  formally  approved  and  full  funding  to  be 
provided  for  the  remainder  of  the  program.  This  is  the  point  at  which  a 
program  manager  would  commit  to  performance,  cost,  and  schedule 
estimates  and  DOD  would  commit  to  provide  that  level  of  resources.  DOD 
later  would  award  contracts  to  product  developers  to  conduct  the  majority 
of  systems  engineering  necessary  for  designing  and  building  the  product.  In 
that  regard,  little  has  changed  under  this  revised  approach.  It  is  still 
difficult  to  know  what  resources  will  be  needed  to  meet  a  set  of 
requirements  before  making  the  decision  to  launch  the  program. 


Pressures  on  Requirements  While  DOD  policy  encourages  evolutionary  acquisition  approaches,  it  will 
Are  Still  Powerful  not  be  successful  if  traditional  incentives  for  setting  high  and  inflexible 

requirements  persist.  Recent  developments  on  the  Global  Hawk  program 
indicate  the  continuing  pressure  programs  face  to  increase  requirements. 
The  initial  Global  Hawk  system  provides  a  capability  for  high  altitude 
reconnaissance  that  the  customer  has  stated  is  acceptable  for  meeting  its 
short-term  needs.  Future  generations  would  add  capabilities  to  match  the 
U-2 — a  manned  reconnaissance  airplane — as  technology,  money,  and  time 
become  available.  The  Air  Force  had  embarked  on  an  evolutionary 
acquisition  that  defines  the  first  generation  of  the  Global  Hawk  based  on 
the  system  already  demonstrated  with  performance,  supportability,  and 
producibility  enhancements.  However,  the  Air  Combat  Command,  the  user 
representative,  indicated  that  it  would  accept  nothing  short  of  the  same 
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capability  found  in  the  current  U-2,  despite  the  customer’s  views.  The 
Command  did  not  have  confidence  that  the  evolutionary  approach  would 
receive  the  support  and  resources  to  attain  the  U-2’s  capabilities  in  future 
generations.  As  a  result  of  pressure  from  the  user  representative,  the  Air 
Force  proposed  a  revised  acquisition  strategy  that  while  still  evolutionary 
in  nature,  accelerated  the  inclusion  of  advanced  technologies  onto  the 
aircraft  to  provide  U-2  capabilities  sooner.  In  December  2000,  the  Deputy 
Secretary  of  Defense  elected  not  to  approve  the  revised  acquisition  strategy 
at  this  time.  While  we  did  not  review  the  revised  approach  in  detail,  it  did 
appear  risk  had  been  added  to  the  program  as  requirements  increased  and 
resources  changed. 

The  Joint  Strike  Fighter  program  is  another  example  in  which  traditional 
incentives  can  result  in  decisions  contrary  to  best  practices.  While  DOD 
has  designated  the  Joint  Strike  Fighter  as  a  flagship  program  for  acquisition 
reform,  it  is  now  4  years  into  its  acquisition  program  and  has  not  yet 
achieved  a  match  between  requirements  and  resources — particularly  in  the 
form  of  the  technologies  needed.  By  best  practice  standards,  none  of  the 
fighter’s  critical  technology  areas  that  the  program  office  has  identified  are 
expected  to  be  at  readiness  levels  acceptable  for  entry  into  the  engineering 
and  manufacturing  development  phase,  which  is  scheduled  for  2001. 
Reminiscent  of  the  Comanche,  numerous  conflicting  demands  that  the 
fighter  achieve  high  performance  and  low  costs  have  resulted  in 
requirements  that  must  be  satisfied  with  several  technical  advances.  The 
requirements  have  also  proven  to  be  relatively  inflexible.  As  we  reported1 
in  May  2000,  delaying  this  phase  of  the  program  until  technologies  are 
mature  would  improve  the  chances  that  the  Joint  Strike  Fighter  will  be 
fielded  as  planned.  However,  despite  not  having  the  requisite  knowledge 
for  the  technologies,  DOD  has  deemed  the  risks  manageable  and  proposes 
to  proceed  with  the  program  as  planned.  Such  a  decision  reinforces 
traditional  incentives  and  increases  the  likelihood  for  future  problems. 


1  Joint  Strike  Fighter  Acquisition:  Development  Schedule  Should  Be  Changed  to  Reduce 
Risks  (GAO/N SIAD-00-74,  May  9,  2000). 
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Conclusions  Managing  the  setting  of  a  product’s  requirements  in  a  way  that  matches  the 

customer’s  needs  with  the  developer’s  resources  is  critical  for  a  successful 
product  development.  This  kind  of  success — delivering  weapons  that  meet 
needs  within  predicted  costs  and  time  frames — is  essential  if  DOD  is  to  get 
what  it  wants  from  its  huge  modernization  investment.  The  best  practices 
for  balancing  needs  and  resources  before  committing  to  a  new  product  or 
weapon  system  development  are  to  (1)  conduct  systems  engineering  to 
illuminate  what  has  to  be  done  to  match  the  wants  with  the  resources; 

(2)  establish  fixed  cycle  times  for  an  initial  product  within  an  overall 
evolutionary  approach  to  foster  flexibility  needed  to  make  key  trade-offs; 
and  (3)  maintain  parity  between  requirement  setters  and  product 
developers  when  translating  customer  needs  into  product  requirements. 
These  practices  provide  for  identifying  gaps  between  wants  and  resources 
as  well  as  solutions — whether  by  trading  off  needs  or  investing  more 
resources — before  setting  requirements  and  starting  development. 

The  environment  for  commercial  products  provides  incentives  that 
encourage  these  practices.  Like  DOD,  commercial  customers  have  an 
initial  advantage  in  that  they  have  high  expectations  of  a  new  product,  they 
do  not  have  unlimited  time  and  money,  and  ultimately,  their  needs  have  to 
be  met.  While  firms  are  motivated  to  offer  a  customer  a  product  that  meets 
these  needs,  they  are  keenly  aware  that  promising  more  than  they  can 
deliver  will  ultimately  disappoint  the  customer.  Consequently,  leading  firms 
consciously  attempt  to  manage  customer  expectations  and  put  themselves 
in  a  good  position  to  negotiate  a  reasonable  set  of  product  requirements. 
They  do  this  in  several  ways.  First,  because  they  are  investing  their  own 
resources  to  develop  the  product,  they  are  at  liberty  to  conduct  systems 
engineering  early — an  investment  they  consider  small  relative  to  the  overall 
cost  of  developing  a  product.  Second,  they  have  a  proven  track  record  for 
delivering  products  on  time  and  making  promised  improvements  to 
succeeding  versions  of  the  product.  Third,  they  play  a  prominent  role  in 
setting  and  agreeing  to  product  requirements.  Together,  these  factors  help 
create  the  credibility,  confidence,  and  trust  that  are  essential  to  maintaining 
the  flexibility  needed  to  match  their  resources  with  the  customer’s  needs 
before  launching  a  new  product  development. 

Within  DOD’s  traditional  acquisition  process  it  is  difficult  to  gain 
knowledge  and  maintain  flexibility  in  requirements  prior  to  committing 
resources  to  an  acquisition  program.  As  a  result,  customer  needs  are 
translated  into  product  requirements  that  often  make  unreasonable 
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demands  on  available  resources.  There  are  several  reasons  for  setting 
weapon  system  requirements  this  way.  Some  are  the  following: 

•  The  mechanics  of  the  program  approval  and  funding  process  force  a 
sequence  of  events  that  keeps  a  product  developer  relatively  uninvolved 
in  shaping  requirements,  compared  with  best  practices,  causing  product 
development  to  begin  with  cost  and  schedule  targets  untempered  by 
systems  engineering  knowledge. 

•  Unique  or  demanding  requirements  can  help  gain  approval  to  fund  a 
new  program  because  aiming  too  low  can  result  in  losing  out  to  other 
programs,  thus  denying  any  new  capabilities. 

•  There  is  some  mistrust  in  an  evolutionary  approach,  specifically 
regarding  whether  future  improvements  will  be  approved;  consequently, 
attempting  to  reach  for  the  full  capability  in  a  single  step  can  be  seen  as 
a  safer  course  of  action. 

•  The  challenging  and  long  process  to  get  requirements  approved  hardens 
requirements  setters  against  trade-offs — in  fact,  the  process  of  getting  a 
requirement  approved  can  actually  encourage  additions  to  gamer 
support. 

•  Unlike  the  commercial  world,  DOD  customers  have  proven  to  be 
tolerant  of  cost  and  schedule  increases  once  a  program  is  underway. 

Seen  in  this  light,  while  the  pattern  of  weapon  system  requirements 
outstripping  planned  resources  is  inefficient,  it  is  not  irrational;  within  the 
DOD  environment,  this  approach  is  successful  in  starting  programs  that 
eventually  provide  superior  capabilities.  This  approach  also  has  negative 
consequences.  First,  the  additional  time  and  money  that  must  be  invested 
after  launch  still  yield  the  same  capability  called  for  by  the  original 
requirements,  effectively  lowering  the  buying  power  of  the  investment. 
Second,  the  unplanned  nature  of  the  additional  investment  generally 
requires  weapon  system  quantities  to  be  lowered  or  money  to  be  taken 
from  other  programs.  Third,  when  a  requirements  setter  is  unwilling  to 
make  trade-offs  before  launch  but  later  elects  to  accept  delivery  delays  to 
meet  those  requirements,  the  implicit  trade-off  is  to  have  the  customer  do 
without  any  of  the  new  capability  for  a  longer  period  of  time.  The  challenge 
in  adopting  best  practices,  therefore,  is  to  make  them  rational — that  is, 
critical  for  success — in  the  DOD  environment.  Meeting  this  challenge  will 
take  changes  in  both  the  mechanics  and  the  incentives  of  the  requirements 
setting  and  program  approval  processes. 

DOD’s  recently  revised  policies  could  make  it  possible  for  a  better 
matching  of  customer  needs  and  developer  resources  before  program 


Page  67 


GAO-Ol-288  Best  Practices 


Chapter  5 

Conclusions  and  Recommendations  for 
Executive  Action 


launch.  While  these  policies  are  sound  and  merit  support,  they  retain  the 
mechanics  of  the  old  policies.  Namely,  requirements  must  be  set  before  a 
program  can  be  approved  and  a  program  must  be  approved  before  the 
money  can  be  obtained  to  pay  the  product  developer  to  conduct  systems 
engineering.  Such  mechanics  deny  the  knowledge  needed  to  match  wants 
with  resources  before  starting  a  program.  Similarly,  the  new  policies  must 
overcome  the  still  extant  pressures  brought  to  bear  on  requirements  setters 
to  aim  high  and  become  inflexible.  It  took  unique  circumstances — starting 
as  Advanced  Concept  Technology  Demonstrations  and  er\joying  the 
personal  intervention  of  top  DOD  executives — to  create  the  incentives  on 
the  Global  Hawk  and  Tactical  Unmanned  Aerial  Vehicle  programs  for 
making  the  trade-offs  needed  to  match  needs  with  resources.  Even  so,  the 
Global  Hawk  is  struggling  with  pressures  to  reverse  some  of  the  trade-offs 
and  raise  requirements.  Other  programs,  such  as  the  Joint  Strike  Fighter, 
have  requirements  that  outstrip  resources,  despite  efforts  to  keep 
requirements  flexible  and  to  treat  the  price  of  the  aircraft  as  a  requirement. 


Recommendations 
Executive  Action 


To  realign  the  mechanics  of  the  requirements  setting  and  program  approval 
processes  to  bring  more  knowledge  into  the  process  of  setting 
requirements,  we  recommend  that  the  Secretary  of  Defense  require  that  the 
systems  engineering  needed  to  evaluate  the  sufficiency  of  available 
resources — knowledge,  time,  money,  and  capacity — be  conducted  in  time 
to  help  identify  and  make  the  critical  trade-offs  that  precede  the 
formalization  of  requirements.  One  option  is  to  allow  the  award  of 
well-defined  systems  engineering  contracts  to  prospective  product 
developers — contractors — before  the  system  development  and 
demonstration  phase. 


To  realign  the  incentives  of  the  requirements  setting  and  program  approval 
processes  with  the  need  to  match  available  resources,  we  recommend  that 
the  Secretary  of  Defense: 


•  Reduce  the  pressures  put  on  user  representatives  to  set  requirements 
high  to  win  the  competition  for  program  approval.  One  way  to  reduce 
these  pressures,  drawing  on  the  experiences  of  the  Tactical  Unmanned 
Aerial  Vehicle  and  Global  Hawk,  is  to  have  higher  level  officials  in  the 
services  and  DOD  decide  on  the  type  of  weapon  system  that  is  needed  to 
meet  a  valid  need  before  the  requirements  setters  begin  detailed  work 
on  framing  a  specific  solution.  Making  such  a  decision  earlier  in  the 
process  would  ease  the  pressure  on  to  set  overly  demanding  and 
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inflexible  requirements  that  will  crush  alternatives  and  win  program 
approval. 

•  Require,  as  a  condition  for  starting  the  system  development  and 
demonstration  phase  for  a  weapon  system — program  launch — that 
sufficient  evidence  exists  to  show  there  is  a  match  between  a  weapon 
system’s  requirements  and  the  resources  the  program  manager  has  to 
develop  that  weapon.  Based  on  our  current  and  past  work  on  the  best 
practices  of  leading  commercial  firms,  there  is  a  key  tool  the  Secretary 
can  use  to  define  what  resources  DOD  is  willing  to  apply— establishing 
limits  on  the  time  it  takes  to  complete  system  development,  such  as  not 
to  exceed  5  years.  Further,  having  a  formal  agreement  among  the 
requirements  setters,  program  managers,  and  resource  providers  on 
development  and  delivery  of  the  required  product  would  emulate  the 
best  practice  of  establishing  accountability  for  subsequent  actions  that 
stray  from  the  agreement. 


Agency  Comments  and  D0D  concurred  with  a  draft  of  this  report  and  its  recommendations  and 
Our  Evaluation  agreed  that  the  requirements  process  needs  to  be  better  informed  by 

'  1 '  ' '  systems  engineering  in  order  to  allow  for  the  timely  leveling  of  user  needs 

and  developer  solutions.  It  noted  that  the  recently  revised  acquisition 
policy  takes  the  first  steps  in  this  new  direction  with  guidance  for 
evolutionary  acquisitions  and  the  identification  of  knowledge  points  in  the 
acquisition  process,  but  agreed  that  more  progress  needs  to  be  made. 
Specifically,  DOD  recognized  that  more  knowledge  in  the  setting  of 
requirements  and  the  potential  ability  of  the  producer  to  meet  those 
requirements  would  provide  a  greater  understanding  of  the  time,  cost,  and 
potential  success  of  a  program.  DOD  also  recognized  that  the  systems 
development  and  demonstration  phase  can  begin  with  a  higher  level 
definition  of  requirements  (that  is,  a  higher  level  than  the  Operational 
Requirements  Document),  which  can  be  used  by  the  requirements-setter 
and  the  program  manager,  after  some  systems  engineering  work  is  done,  to 
facilitate  trade-offs  earlier  and  relieve  some  of  the  pressure  on  Operational 
Requirements  Document.  Finally,  DOD  stated  that  the  Acquisition  Program 
Baseline  should  require  signatures  from  both  the  user  and  the  resource 
sponsor  prior  to  program  initiation  signifying  agreement  that  there  is 
sufficient  evidence — such  as  determined  by  systems  engineering, 
demonstration  of  technology  maturity,  and  adequacy  of  funding — that  a 
match  between  requirements  and  resources  has  been  made. 
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TECHNOLOGY 


PRINCIPAL  DEPUTY  UNDER  SECRETARY  OF  DEFENSE 
3015  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3015 


FEB  8 


Ms.  Katherine  U.  Schinasi 
Director,  Acquisition  and  Sourcing  Management 
U.S.  Genera!  Accounting  Office 
Washington,  DC  20548 

Dear  Ms.  Schinasi: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  “BEST  PRACTICES:  Better  Matching  of  Needs  and  Resources  Will  Lead 
to  Better  Weapon  System  Outcomes,"  dated  December  22,  2000  (GAO  Code  707454/OSD 
Case  3022).  The  Department  concurs  with  the  GAO  draft  report  and  agrees  with  the 
recommendation  of  the  need  to  level  requirements,  resources,  and  time-lines.  The  recently 
revised  5000  process  helps  in  that  regard  by  fixing  the  development  side  of  the  leveling  - 
evolutionary  acquisition,  identifying  knowledge  points,  etc. 

The  GAO  is  also  correct,  however,  that  we  still  do  this  within  the  context  of 
requirements  to  resources  to  systems  engineering.  The  Department  needs  to  correct  that  to 
allow  the  final  setting  of  requirements  and  resources  to  be  completed  only  after  some  systems 
engineering  is  done.  This  way,  the  program  manager  has  a  greater  ability  to  negotiate  with  the 
requirements  and  resource  communities.  The  first  steps  in  this  new  direction  have  already 
been  taken  with  the  recent  5000  rewrite,  but,  as  GAO  points  out,  more  progress  needs  to  be 
made.  The  Department  welcomes  the  GAO’s  recommendations  and  our  specific  responses  to 
these  recommendations  are  included  herein  as  an  enclosure. 

Thank  you  for  the  opportunity  to  review  and  comment  on  the  report.  The 
professionalism  and  the  level  of  cooperation  between  my  staff  and  yours  are  greatly 
appreciated  and  we  look  forward  to  working  with  your  staff  again  in  the  future. 

Sincerely, 


DaveOtor 


Enclosure: 
As  Stated 
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GAO  DRAFT  REPORT  -  DATED  DECEMBER  22,  2000 
GAO  CODE  7074541  OSD  CASE  3022 

“BEST  PRACTICES:  BETTER  MATCHING  OF  NEEDS  AND  RESOURCES  WILL 

LEAD  TO 

BETTER  WEAPON  SYSTEM  OUTCOMES" 

DEPARTMENT  OF  DEFENSE  COMMENTS  TO  THE  RECOMMENDATIONS 

RECOMMENDATION  1 :  The  GAO  noted  that  to  realign  the  mechanics  of  the 
requirements  setting  and  program  approval  process  to  bring  more  knowledge  into  the 
process  of  setting  requirements,  the  GAO  recommended  that  the  Secretary  of  Defense 
Now  on  pp.  1 1  and  67.  require  that  the  systems  engineering  needed  to  evaluate  the  sufficiency  of  available 

resources-knowledge,  time,  money,  and  capacity--be  conducted  in  time  to  help  identify 
and  make  the  critical  tradeoffs  that  precede  the  formalization  of  requirements  The  GAO 
noted  that  one  option  the  Secretary  could  consider  for  advancing  the  timing  of  systems 
engineering  is  allowing  the  award  of  well-defined  systems  engineering  contracts  to 
prospective  product  developers-contractors-before  the  system  development  and 
demonstration  phase,  (p.  51/Draft  Report) 

DOD  RESPONSE:  Concur.  The  Department  agrees  that  the  requirements  process 
needs  to  be  informed  through  systems  engineering  in  order  to  allow  for  the  leveling  of 
,USeI?^S  and  devel°Per  solutions.  In  fact,  our  changes  to  the  requirements  process 
(in  CJCSI 31 70.01  A)  and  the  acquisition  process  (in  DoDI  5000.2)  support  this 
recommendation  by  encouraging  the  development  of  technologies  and  components 
before  the  beginning  of  system  development  and  the  creation  of  an  Operational 
Requirements  Document  (ORD).  Also,  our  current  Analysis  of  Alternatives 
methodology  includes  systems  engineering  tasks  such  as  requirements  and  functional 
analysis/allocation.  However,  more  needs  to  be  done.  For  example,  systems 
development  and  demonstration  can  begin  with  higher  level  "requirements"  that  define 
caPa^^^ef  (e-9-»  a  statement  of  operational  capabilities  approved  by  the 
JROC)  then  negotiate  schedule,  funding,  and  performance  requirements  through 
interaction  with  the  program  manager  and  the  requirements  generator  so  that  those 
capabilities  can  be  translated  into  firm  requirements  that  would  be  captured  in  an 
operational  requirements  document.  We  recognize  that  more  knowledge  in  the  setting 
of  requirements  and  the  potential  ability  of  a  producer  to  meet  those  requirements 
would  provide  a  greater  understanding  of  the  time,  cost,  and  potential  success  of  a 
program. 

RECQMMENDATION  2:  The  GAO  noted  that  to  realign  the  incentives  of  the 
requirements-setting  and  program  approval  processes  with  the  need  to  match  available 
resources,  the  GAO  recommended  that  the  Secretary  of  Defense  reduce  the  pressures 
approval representatives  t0  set  requirements  high  to  win  the  competition  for  program 
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Now  on  pp.  1 1  and  67. 


Now  on  pp.  1 1  and  68. 


The  GAO  noted  that  one  way  to  reduce  these  pressures,  drawing  on  the 
experiences  of  the  Tactical  Unmanned  Aerial  Vehicle  (TUAV)  and  Global  Hawk,  is  to 
have  higher  level  officials  in  the  Services  and  DOD  make  the  decision  on  the  type  of 
weapon  system  needed  to  meet  a  valid  need  before  the  requirements  setters  begin 
detailed  work  on  framing  a  specific  solution.  Making  such  a  decision  earlier  in  the 
process  would  ease  the  pressure  on  setting  overly  demanding  and  inflexible 
requirements  that  crush  alternatives  and  win  program  approval,  (p.  51/Draft  Report) 

DOD  RESPONSE:  Concur.  The  Department  believes  the  recommendation  can  be 
accomplished  by  using  a  requirements  document  such  as  the  mission  need  statement, 
that  defines  needed  capabilities  that  would  be  translated  into  an  operational 
requirements  document  (ORD)  after  some  systems  engineering  work  is  done  (as 
discussed  in  the  response  to  recommendation  1).  Use  of  a  higher  level  more  general 
requirements  document  to  facilitate  tradeoffs  earlier  would  improve  this  process  and 
relieve  some  of  the  pressure  on  the  ORD.  Weighing  the  advantages  and 
disadvantages  of  trading  cost  and  performance  to  meet  the  general  requirements  can 
be  guided  by  the  principles  of  "Cost  as  an  Independent  Variable." 

RECOMMENDATION  3:  The  GAO  noted  that  to  realign  the  incentives  of  the 
requirements-setting  and  program  approval  processes  with  the  need  to  match  available 
resources,  the  GAO  recommended  that  the  Secretary  of  Defense  require,  as  a 
condition  for  starting  the  system  development  and  demonstration  phase  for  a  weapon 
system-program  launch-that  sufficient  evidence  exists  to  show  there  is  a  match 
between  a  weapon  system's  requirements  and  the  resources  the  program  manager  has 
to  develop  that  weapon. 

Based  on  the  GAO  current  and  past  work  on  the  best  practices  of  leading 
commercial  firms,  there  is  a  key  tool  the  Secretary  can  use  to  help  define  what 
resources  the  Department  is  willing  to  apply-establishing  limits  on  the  time  it  takes  to 
complete  system  development,  such  as  not  to  exceed  5  years.  Further,  having  a  formal 
agreement  between  the  requirements  setters,  program  managers,  and  resource 
providers  on  development  and  delivery  of  the  required  product  would  emulate  the  best 
practice  of  establishing  accountability  for  subsequent  actions  that  stray  from  the 
agreement,  (p.  51 /Draft  Report) 

DOD  RESPONSE:  Concur.  The  Department  agrees  that  there  should  be  an 
agreement  between  the  user,  the  developer,  and  the  resource  sponsor  in  which  all  sign 
up  to  the  requirement,  time-line,  and  costs.  The  need  to  match  requirements  to 
available  resources  is  obvious  and  essential  for  program  success.  Currently,  the 
Acquisition  Program  Baseline  (APB)  is  used  as  an  agreement  within  the  acquisition 
chaln-of-command  on  resources,  schedule,  and  performance.  Both  the  user  and 
resource  sponsor  concur  in  that  baseline.  The  user  and  resource  sponsor’s  agreement 
could  be  made  more  formal  with  the  requirement  for  their  signatures  on  the  APB. 
Further,  the  APB  should  contain  sufficient  evidence  that  a  match  between  requirements 
and  resources  has  been  made,  such  as  demonstration  of  technology  maturity,  results  of 
systems  engineering,  and  the  adequacy  of  available  funding  to  cover  acquisition  costs. 
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The  recently  revised  CJCS1 3170.01  A  and  DoDI  5000.2  both  require  that  costs 
be  addressed  as  part  of  the  development  of  the  operational  requirement.  This  was 
done  to  provide  a  better  match  between  resources  and  requirements.  We  have  not 
established  a  hard  and  fast  rule  about  length  of  time  for  system  development,  because 
length  of  time  is  very  dependent  on  the  type  of  system.  However,  the  USD(AT&L) 
memorandum  forwarding  the  revised  DoD  5000  documents  does  address  a  planning 
horizon  of  five  to  seven  years  for  translation  of  a  mission  need  into  a  fielded  system. 
Also,  we  are  tracking  cycle-time  as  part  of  our  GPRA  metrics  with  an  internal  stretch 
goal  of  reducing  cycle-time  in  general  from  an  average  of  132  months  to  66  months. 
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The  first  copy  of  each  GAO  report  is  free.  Additional  copies  of 
reports  are  $2  each.  A  check  or  money  order  should  be  made  out  to 
the  Superintendent  of  Documents.  VISA  and  MasterCard  credit 
cards  are  accepted,  also. 

Orders  for  100  or  more  copies  to  be  mailed  to  a  single  address  are 
discounted  25  percent. 

Orders  by  mail: 

U.S.  General  Accounting  Office 
P.O.  Box  37050 
Washington,  DC  20013 

Orders  by  visiting: 

Room  1100 

700  4th  St.  NW  (corner  of  4th  and  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC 

Orders  by  phone: 

(202)  512-6000 
fax:  (202)  512-6061 
TDD  (202)  512-2537 

Each  day,  GAO  issues  a  list  of  newly  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any  list 
from  the  past  30  days,  please  call  (202)  512-6000  using  a  touchtone 
phone.  A  recorded  menu  will  provide  information  on  how  to  obtain 
these  lists. 

Orders  by  Internet: 

For  information  on  how  to  access  GAO  reports  on  the  Internet, 
send  an  e-mail  message  with  “info”  in  the  body  to: 

info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  home  page  at: 
http://www.gao.gov 


Contact  one: 

•  Web  site:  http://www.gao.gov/fraudnet/fraudnet.htm 

•  e-mail:  fraudnet@gao.gov 

•  1-800-424-5454  (automated  answering  system) 
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